Before we get to the exciting conclusion of Star Fox Assault, let's take a step back and look at THE most requested Star Fox Assault cutscene. I figure it's as good a Krystmas present as any!

This is the now-famous Honeymoon cutscene, where Fox and Krystal meet up with a grown-up Tricky after fighting off the Aparoid threat on Sauria together. Tricky mades a casual mention that Fox and Krystal should come back for their honeymoon. Fox reacts with embarrassment, while Krystal laughs at his plight.

As some of you know, Star Fox Assault features several pre-rendered cutscenes. There are ten, I think. Even though the cutscenes appear to use the in-game engine, for whatever reason, they implemented these scenes as 30 frame per second, heavily compressed video, at a mere 640 by 336 resolution. But they did even worse, because that 30fps video is set to run at 60fps, doubling each frame. Except for some parts of the video that is running at less than 30fps, yet still displayed as 60fps video. Then, the video got vertically stretched to make it slightly taller. The compression they used left very big artifacts on the screen too.

What can we do about this? First, instead of recording the video off the screen, I instead pulled individual frames of video as textures. This allowed me the pure data of each video frame: no stretching, skips, or duplicates. This had the added bonus of showing more of the video, because some sections during fade-outs are handled by the game, not by the video frames.

Next, I tried all sorts of different high quality upscaling techniques. I tried every scaling filter ffmpeg has trying to find the best one, which turned out to be 'bicublin'. I also researched some paid upscalers. Ultimately, thanks to some advice from juiceps, I rediscovered a project from 2015: waifu2x.

Waifu2x is a GPU accelerated image upscaler that uses Neural Networks to increase the resolution of an image, particularly drawn images, without losing detail. The idea started with a Chinese research paper, Image Super-Resolution Using Deep Convolutional Networks. From there, it was implemented as open source software by nagadomi from Japan. There is even a project for automating and bulk-processing the upscaling technique with fully customizable settings with the project waifu2x-caffe. Visually, the technique has the effect of detected edges, and redrawing them as sharp lines, then taking everything else and smoothing the gradient of color so that they appear clean and free of artifacts. Waifu2x is FAR better than any other upscaler I've ever seen.

Here's what the process was like:

1. Record the frames using frame dumping (also record the audio and the cutscene normally as a video to match against
2. Use ImageMagick to bulk crop out the subtitle section, leaving the visual frames
3. Run all the cropped frames through waifu2x-caffe, using the upRGB algorithm at Level 2 strength, and scaling the image up by 600% to make it 4K width (this process takes approximately 6 hours per 1 minute of footage)
4. Use FFMPEG to convert the frame images into a 30fps video at 4K resolution
5. Import the video into Final Cut Pro and edit it using the original recordings as a guide, doing any necessary frame speed retiming
6. Finally export as a h.264 high quality video

Needless to say, this is a tremendous amount of work for each video. That's why they take so much longer than the other kinds like briefings and other scenes. But I have it figured out now, so future videos of this type should be coming more quickly.

For the 4K version, I removed the subtitles, since there wouldn't be much room for them anyway. However, I also made an original version, below. It uses the frames at exactly the same resolution as they exist in the game data, so it's maximum resolution is 640 by 480. I did record slightly higher resolution subtitles, though. Also, the frame timing and fades are better here than in the real game video. But put this video in full screen, and then compare it to the 4K version. The difference is staggering!

I will post the Japanese version of these scenes soon. But for now, I'm going to take a break for the rest of this week to celebrate Christmas and New Years. I hope you will also take some time to relax, hang with family, and reflect on the year past. See you in 2018!