Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
MrFlibble

A simple method of scaling up 2D sprites with ESRGAN

Recommended Posts

I've been experimenting with neural network upscaling of various old video game art assets for quite some time already, and recently found a way to produce quite acceptable results using ESRGAN without much manual editing, proprietary software or complex procedures. Here's a sample with some sprites from Strife scaled to 2x the original size (zoomed 2x mode with nearest neighbour for better visibility):

Below is a step-by-step description of how to produce similar results.

 

I. Setting up.

Spoiler

 

1. What you'll need:

2. I will not explain here how to set up ESRGAN; there's a useful guide here, and also a front-end which however I never used so I can't really recommend it. The following steps assume that you have ESRGAN installed and understand its operation. Note that if you do not have an NVIDIA card or cannot install CUDA you can stick with CPU processing although that will greatly increase performance time.

 

3. You will need to produce interpolated models from the ones you downloaded from patrikspacek. First, put the models mymanga109_250000.pth and cartoonpainted_80000.pth into the models sub-folder of the ESRGAN main folder. Then open the file net_interp.py that comes with ESRGAN in any text editor of your choice, and find the line net_PSNR_path = and alter the following segment (replace model names) to:


net_PSNR_path = './models/RRDB_ESRGAN_x4.pth'
net_ESRGAN_path = './models/mymanga109_250000.pth'

Then run net_interp.py like this:


python3 net_interp.py 0.8

That will produce a new model called interp_08.pth. You can rename it to something like MyManga109_08.pth. The do the same with the cartoonpainted_80000.pth. (The above sample images are produced with the MyManga109 interpolated model).

 

That's it, you're ready to go. To use the new interpolated models, run test.py with the respective model name as an argument. The modified version of test.py I liked to above will append the model name to the output file name.

 

II. Preparing images for processing.

patrikspacek's models are versatile enough to handle raw 8-bit images more or less properly but I discovered that the following pre-processing method yields noticeable improvement that is quite crucial when it comes to sprite edges.

Spoiler

 

1. Assemble a sprite sheet that you want to upscale. Very large images are not recommended for processing, but individual sprites would be too cumbersome to process. In any case, take care that the sprites do not touch the edges of the image and there is at least one pixel thick border between any part of the sprite and adjacent sprites, and image edges. Preferably use a dark background colour, most crucially it must be a colour that is not similar to any colour used in the sprite itself.

 

2. Open the resulting image in xBRZ Scaler Test and scale it up to 4x the original size using the xBRZ scaler. Don't forget to turn off the "Show scaler name" option. Save the result as a separate image (I use a -z postfix but it can be anything). You can batch-process images using the script by Phredreeke available here.

 

3. Open the scaled-up sheet in GIMP. Apply Gaussian blur at 0.5 pixel radius. Scale the image down to the original resolution using Sinc interpolation. Alternatively, you can apply blur at 1.0 pixel radius and use Bicubic interpolation instead. Bicubic results in a sharper scaled-down image, so blurring somewhat compensates for it. This sharpness will translate to the ESRGAN upscale result so you can change this variable to get sharper upscales if you want. Save the result.

 

III. Put the pre-processed image(s) into the LR folder of the ESRGAN folder and run test.py with one of the interpolated models. I have generally found that the MyManga109 250k produces quite good results with various video games' graphics (here are some results of scaling up complete screenshots).

 

IV. Now let's produce the masks for the upscaled sprites. This is made easy because both patrikspacek's MyManga109 and Cartoon Painted models can handle sprite edges very well on their own, especially in pre-processed images.

Spoiler

 

1.Let's first process the mask for the 4x upscale, which is the easiest. Open the ESRGAN output file (if you used the modified test.py it should be called something like myimage_MyManga109_08.png depending on how you named the interpolated model and the input image) in mtPaint and save a copy. I usually add the KN postfix because what we're going to do here is apply Kuwahara-Nagao blur (available in the Effects menu) at 2 pixel radius.

 

2. After the KN blur, you need to convert the image to indexed palette mode. In mtPaint, you can load a palette straight from another image. There are two ways you can do that:

 

a) take a screenshot of the game (e.g. Strife) in DOSBox, preferably at default brightness. That will give you the entire palette at once.

 

b) However, I intuitively prefer a different method. Remember the sprite sheet you assembled? Open that up in mtPaint and select Image -> Convert to Indexed. Unless you put too many different sprites on the sheet it will offer you to create a palette that is an exact match of the present colours (if the number of colours is 256 or less). Save the result.

 

3. Now back to the Kuwahara-Nagao 4x image, load the indexed palette using the palette menu and select the image produced by step 2b. This will allow to convert the scaled up result to exactly the same colours as the source image. Now, select Convert to Indexed, and pick the following options:

  • set palette to current palette
  • set dithering mode to none
  • click options and set colour space to RGB (default is sRGB)

Click OK to convert the KN image.

 

4. Open both the original ESRGAN 4x result and the KN indexed image. With the latter active, press Shift+O to activate GIMP's select by colour tool, and click anywhere on the background. This will only select the background colour (purple in the examples above).

 

5. Right-click the selection and from the dropdown menu pick Selection -> Grow. Increase the selection by 1, then copy the result over the ESRGAN 4x image. Export the combined image.

 

6. Now you have the sprite sheet with clean sprite edges. If you want you can palettise the result following the same steps as desctibed in paragraphs 2 and 3. To get a transparent background, open the image in GIMP, first select Layer -> Add alpha channel, then again use the select by colour tool to pick out the background and simply delete it.

 

V. If you want to get a 2x upscale, take the ESRGAN 4x output, make a copy and scale it down to 2x. The easiest is to pick any version of Bicubic that mtPaint offers. Then, make a copy and apply Kuwahara-Nagao blur, but this time at 1 pixel radius, and also select the option to protect details. After that, take the same steps as in 2-6 in section IV above. This is how I produced the Strife images shown at the top of the post.

 

More examples: Doom, Hexen, Daggerfall (4x).

Share this post


Link to post

Very interesting read and a surprisingly high amount of steps to achieve these results. But those Strife sprites look really great. It seems that their original pixel art style lends itself nicely for upscaling. 

 

Great work, please continue and don‘t give up. 

Share this post


Link to post
On 5/29/2019 at 4:29 PM, elend said:

But those Strife sprites look really great. It seems that their original pixel art style lends itself nicely for upscaling.

It isn't really related to the original art; it's how the model is trained. There are newer models compared to what I have used, which produce very good results, and without the need to pre-process images as I described in my previous post.

 

Here's what I got from an interpolation of Fatality, Rebout and DeToon models (details here):

image.png.92cbe88d3ebc424896bad1f7ff73f654.png

And here's the same model applied to pre-rendered sprites from Doom 64:

image.png.74fe46ef6448487ad99348a2e0eb07d1.png

Share this post


Link to post

Wow, that does look very clean and awesome. Great stuff.

Share this post


Link to post

I fiddled with ESRGAN on Doom 64 a little bit a while ago and didn't like the results, but it looks like the models have improved quite a bit! I should give it a go again. 

Share this post


Link to post

Beautiful Strife art, not so good Doom 64 result.

 

Strife stuff just needs some small touchups like to the gun, muzzleflash, or that dudes neanderthal face.

Doom 64 would be better remaking the models and doing new captures from those.

Share this post


Link to post
On 8/29/2019 at 3:37 AM, wheresthebeef said:

Beautiful Strife art, not so good Doom 64 result.

What do you think is wrong with the D64 upscales? Usually pre-rendered sprites seem to come out somewhat better that hand-drawn art when neurally upscaled, but of course that varies.

Edited by MrFlibble

Share this post


Link to post
15 hours ago, MrFlibble said:

What do you think is wrong with the D64 upscales?

 

They have a 'stony' look and feel to them, instead of living flesh. It also hasn't filled in the implied details as well as with other sprite sets (notably the Imp's teeth, Mancubus mouth).

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×