Doom Neural Upscale 2X [v 1.0]

Waifu turned the 3 red things on the upper right to curves?!

9 hours ago, elend said:

Yes, they do. In my opinion they retain the "hand pixelled" style better, than the "oily" waifu upscales. That's exactly what got me excited about this in the first place. Although, I have to admit, that the sprites themselves look quite okay with waifu.

8 hours ago, VGA said:

Waifu turned the 3 red things on the upper right to curves?!

That's the result of prior image softening and not an inherent feature of waifu2x scaling per se (BTW, waifu2x is also based on neural networks so it's better to refer to the initial approach listed here as NVIDIA scaling or something). Softening works very well with some kinds of art but apparently not with others. However this is completely optional, here's the same texture set processed by waifu2x without softening or any other pre-processing (also not edited in any way afterwards):

Does it look better to you?

Anyway, I was thinking about the whole thing and came to the conclusion that there is no easy way to upscale low-res assets because of the inherent limitations of the low resolution. It's not just about limited detail, there seems to be some idiosyncrasies that are not easily avoided no matter how you scale up the images.

For example, you can notice some pretty odd changes to the Zombieman head's shape in the frontal view firing animations. I'm guessing this is simply due to how the low-res art worked, but after scaling it up it looks weird.

I believe that the scaled up sprites can be used in some kind of "crispy" mod which retains the original look and feel of the art while making it sharper. I think it's definitely better than using other types of sprite scaling like xBRZ which essentially smooth pixel edges but don't otherwise alter an image in any way (e.g. a pixel-thin line will become two pixels thick at 2x scaling with xBRZ, without any colour transitions or the like). For this purpose, I don't believe that much editing is needed, at least for the sprites. Otherwise the results of scaling could serve as a basis or reference of a complete graphics overhaul, but that will require much more work than just editing a few pixels here and there, and will end up as original art while straightforward scaling at least remains more or less "true" to the source material. (It's not like an artist's work can't be true to the originals, what I mean to say is that the art which is automatically scaled is, in my opinion, basically the same stuff but smoother, while a creative element by an artist produces something definitely different.)

UPD: I think I indeed placed undue faith in the softening method. Here's the same set of character sprite samples scaled up with waifu2x without prior softening (not converted to the Doom palette):

This time I used the old xBRZ mask method as described here. There's definitely more detail preserved this way.

First of all; Thank you really, really much for your work. Much appreciated!

Second: So, you agree, that a more hand pixelled look would be preferable, right? Your second examples without image smoothing look at least a bit better to me, than the smoother versions. On that note: Did you think about "sharpening" (somewhere in the process)? Maybe that could help as well, to give them that last "punch" to the details.

Third: Really, your last method is actually quite okay already and I would love to have all those assets in these versions. Nobody said it would be perfect or easy. I just think your current process is already quite an improvement, to the really lowres original Doom textures. I would take it upon myself to manually edit them and - maybe in a year - we could make a .wad out of this. :p I am just not good at all this texture sorting, naming and lumping stuff. T_T

Fourth: If somebody is doing this crazy thing, wouldn't Smoothdoom be the only way to go as a basis?

Fifth: Christ, the Imp is jacked!

A quick test with sharpening your result and Doom paletting it.

3 hours ago, elend said:

So, you agree, that a more hand pixelled look would be preferable, right? Your second examples without image smoothing look at least a bit better to me, than the smoother versions. On that note: Did you think about "sharpening" (somewhere in the process)? Maybe that could help as well, to give them that last "punch" to the details.

This is probably a good time to elaborate on how waifu2x works. If you scale up raw sprites against a solid colour background the result will be bad (all images below zoomed at 2x using nearest neighbour for better visibility):

The sprite edges are not processed at all (for a more extreme example check this).

At the same time waifu2x is rather good at resizing photos and 3D rendered images, which led me to believe that it can't handle sharp contrasts between pixels well. After trying out several methods of softening input images I came up with the "xBRZ blur" I described above. If you process a sprite this way and then scale with waifu2x you'll get this:

Now the outline of the sprite is much smoother, at the same time the entire image has become blurrier, with some detail faded (e.g. the folds of the trousers have become less pronounced, you almost can't see the right side of the helmet's breather etc.). You can sharpen the result but as far as I can tell it won't quite match the non-softened image. While sharpening before scaling up kinda defeats the purpose of softening in the first place.

In addition to this, scaling against a pink background results in odd colours at the sprite's edges, such as the visible green at the top of the helmet in both images above, which shouldn't be there. The only way to get rid of this is to process a second, set of  the same sprites against a neutral grey background (again, as I described above), then extract the pink mask from the first image and paste it onto the second one.

As for the softened versus non-softened sprites, the only way to achieve middle ground I can think of ATM is to only select parts of the non-softened sprite that are not near the edges and paste them on top of the softened one:

However you can still see there are some problematic pixels on the rifle's muzzle for example.

This method would involve processing three separate images, even though perhaps this could be simplified a bit if you replaced waifu2x-created mask with an xBRZ-created mask — personally I haven't noticed as much difference between the two, at least for shapes like the sprites above.

I did try pasting an unaltered sprite over a softened version to scale up this combination, but it didn't work out well because edges get processed properly only if there's mutual blurring of both the pixels on the edges and the neighbouring pixels of the background.

I do understand, what you mean. Never mind the edges. Even if they are as "bad" as in the sharpened version, I posted, this would be the easiest thing to fix, imho. I think the most important thing would be to keep the results as crisp and "pixely" as possible, without turning them into mush.

I hope no one minds me saying so, but as an observer with no wad/mod experience i would very much like to see at the very least the sprites created from here as something usable, because they look very good imo

The Waifu2x and other up-scaling options are better than nothing. I have always wondered if sprites were remade in SVG (scale-able vector graphics). It would take more effort than say a manual 2x scaling, but would technically be infinite resolution. I notice the upscalers in gzdoom weren't bad, so SVG would be a lot better. But not many people are used to making vector graphics and those who are typically are busy with other stuff. Since I am not doing it myself I can't complain either lol.

I am doing a lot of vector graphics actually. If we start remaking the sprites in vector and using this as a base, the handpixelled look will most probably be gone. And along with it, the whole reason for this topic: upscaling with the least possible effort.

I can not let it go and tried to optimize the doomguy sprite. Which means, I added more details, sorted some pixels and tried to generally improve areas, where waifu just left a blurry mess. Unfortunately I still don't really know what's going on around that rifle area. :(

Yet, the difference is striking, especially compared to the original sprite but also compared to the original waifu result. Now imagine a treatment like this done to all the sprites. All the frames. All the rotations. :D

Edit: Oh and of course there are some messed up pixels at the lower right edges, which I didn't see in time.

On 10/7/2018 at 3:35 PM, elend said:

So, you agree, that a more hand pixelled look would be preferable, right? Your second examples without image smoothing look at least a bit better to me, than the smoother versions.

I have a long-standing interest in the different ways that low-resolution graphics from DOS games could be scaled up, however for a long time I did not give a proper thought to the philosophical and artistic sides of the question, assuming that simply smoothing input would be enough. Yet while the softened waifu2x approach can produce some really good results (this example combines a pre-rendered background and hand-painted figures, but they are rather large), it doesn't always work well for small sprites, regardless of how they were originally produced, or textures.

There's also no clear answer to the question what approach would be more acceptable. An appeal to established industry practice would be a good idea, but I'm only aware of a handful of FPS games with sprite-based characters that received official high-res updates, which includes the Macintosh version of Wolfenstein 3-D, the XBLA version of Marathon 2, and Doom 64. All these had completely new sprites that were not produced by scaling up their original low-res counterparts. If we broaden the scope and take any sprite based game into account, the only example I can think of where DOS version sprites were indeed scaled up is the Macintosh version of Warcraft. I'm not sure if they used any scalers or if the sprites were all processed manually, but they were definitely edited to add detail at least to the buildings, although you can still see the large pixels here and there (it is my understanding that the developers had a tight schedule to complete the Mac port).

I wonder if using dithering would contribute to the pixellated look that you're interested in? Here's the same texture set as above, no prior softening, converted to the Doom palette using Stucki dithering in mtPaint:

36 minutes ago, elend said:

I can not let it go and tried to optimize the doomguy sprite. Which means, I added more details, sorted some pixels and tried to generally improve areas, where waifu just left a blurry mess.

Actually that's pretty good! Did you do this entirely from scratch using waifu2x scaling for reference, or edited the waifu'd sprite?

39 minutes ago, elend said:

Unfortunately I still don't really know what's going on around that rifle area. :(

If it is the central part that you're talking about, I think it's a part of Marine's belly showing through (apparently it's bare as shown on the cover art/title screen), and the black thing to the right is the rifle's magazine.

Would it help if you were working with sprites scaled to 4x the original size?

For this set I employed the usual procedure of two images — one for the mask and another with background grey for proper colours at edges — but I also only applied softening to the edges of the sprites to get at least most of each sprite sharper than with the original method.

This does not in any way improve the fact that at 4x the sprites look straight ugly and messy, but at least this could serve as a base for editing them into proper images I suppose.

That Warcraft example you posted does indeed look very good. I think there is a certain resolution threshold a graphic has to surpass to be able to be decently upscaled. That Doomguy sprite is just veeery tiny, of course the upscaled result will look messy and blurry. That's just what we have to live with.

Quote

There's also no clear answer to the question what approach would be more acceptable.

Of course not, since to my knowledge this is usally not done at all. The example of Wolfenstein for the Mac is a good one, since the sprites are entirely redrawn. There is just no other way, especially not "back then". So, for us the only question is: What is the most feasible (in terms of amount of work) and most good looking way? And here the modern upscaling really gave me a bit hope, since the results are quite okay by now. Especially the results of the thread opener, which still look the best to me. Specifically the textures. I am sad, that he doesn't look into the DW forums anymore and I don't know if this is still an ongoing project or not. :(

Quote

Actually that's pretty good! Did you do this entirely from scratch using waifu2x scaling for reference, or edited the waifu'd sprite?

This is just an edit of your waifu result. My thinking was like this: Let waifu do most of the work, then manually retouch them where needed. Turns out there is quite some retouching needed. For this Doomguy I needed about 2 - 3 hours. That's just too much time to apply to all the sprites and rotations. :((( Not to mention that a project like this should probably be based on smooth doom as well, which just ups the amount of work one has to do. I am now at the point, where I think this is really hopeless. At least to have the quality I envision. That Doomguy sprite I just edited for fun, to see what something like that could look like. It's just one sprite of many, who knows if I am able to edit an Imp or a Spider Mastermind with the same quality.

Maybe the original poster of this thread will update us on his work.

26 minutes ago, elend said:

So, for us the only question is: What is the most feasible (in terms of amount of work) and most good looking way? And here the modern upscaling really gave me a bit hope, since the results are quite okay by now. Especially the results of the thread opener, which still look the best to me. Specifically the textures.

I just examined some of the scaled up textures more closely and I see what you mean. NVIDIA's neural network keeps the pixels sharp while waifu2x blurs the small details, making the result completely unlike pixel art. This is bad. I tried sharpening the input textures but blurring still occurs. There seems to be no way around this so far.

However, at least theoretically it should be possible to create a new network trained to produce results similar to NVIDIA's, by training it on exactly the textures that hidfan posted.

Just use the Nvidia thing? For all Doom resources?

I will actually try this out for myself. Just have to get beta access first.

EDIT: And there are no more beta slots available. What a pity. Guess, I'll have to wait till they finish the beta phase. :/

I was thinking about how to imitate the GameWorks Super Resolution look of the scaled up sprites, or at least mitigate the blurring introduced by waifu2x, here are some results:

Same stuff zoomed up 2x by using nearest neighbour for better visibility:

What I used here is scaled the image 4x using waifu2x and then applied a filter called Rock I found in IrfanView. The result was scaled back to 2x using Sinc3 in GIMP.

To avoid colour distortions at the edges I thickened the outline of each sprite by one pixel like this before scaling in waifu2x:

The edges are softened with xBRZ but the sprites in the centre are original quality.

I also processed another image using the regular technique to get the mask which I applied to the sprites with the rock filter. The final result was then converted to the Doom palette.

Can you put out a full set for one enemy?

Here you go:

The only thing different from the above is that I only used the smoothed version for applying the rock filter, as the original version gets too sharp.

It seems you underestimated my laziness. I meant a wad that I can load, not a spritesheet :D

So, this looks definitely better than the rather blurry first attempts. I do like this indeed. How about textures, do they improve as well? I think this solution seems really okay and maybe we should just try a bunch of textures and sprites and see how they look ingame.

I did apply for beta access to that Nvidia Gameworks thingy, but I haven't heard from them yet. At first I thought there's no beta access anymore, but now I am not sure. Their website is anything but user friendly.

Yes, I did a couple of textures to compare with the ones made by hidfan:

https://imgur.com/a/8bb0gIX

https://imgur.com/a/GkqB3dK

All textures zoomed 2x with nearest neighbour. In each set, the top image is GameWorks 2x as posted by hidfan, the middle is waifu2x smoothed by xBRZ, first scaled to 4x, applied the Rock filter and then scaled to 2x with Sinc3, and the bottom is pure xBRZ to 4x, applied the Rock filter and scaled to 2x with Sinc3.

As far as I can tell the Rock filter is some kind of emboss effect, as a result you'll notice slight a shift to the right when comparing the textures. Perhaps a more subtle emboss effect in another editing programme (e.g. GIMP) would sharpen accentuate the edges the same way without this shift but I haven't tried anything like this yet.

I think we are getting somewhere actually. Your recent examples look quite good now, even though I would prefer a more subtle approach regarding the "rock" filter as well. But, I like those results much better than the usual options in GzDoom like interpolation or those weird filters you can switch on.

Yet, it still amazes me, how perfect the Nvidia samples look and how hard it seems to reach similar results. Oh @hidfan where are you? :___(

Some random notes that might help you guys:

1. The background matters when doing sprites. These advanced resizing filters do their magic by comparing the current pixel with neighboring pixels, to try to determine if the current pixel is part of a curve, a straight line, or just a detail. So, carefully choosing the background will have a drastic effect on the edges of your sprite.
2. Sprites are very different than textures. Sprites are representing an object with lots of depth, and with shadows and highlights "baked in". Textures, on the other hand, generally represent a nearly flat surface, with some exceptions. I'm no artist, but I've noticed that artists draw sprites very differently. My point is that the resizing filters can rely on this baked in shadowing in sprites, and it makes sense that you might have to approach textures differently.
3. The general problem with upsizers are that the original detail is gone. The upsizers you're currently using are almost performing miracles, in that they are guessing what pixels should be there...but it's only a guess.

Finally, I must say that the results you have provided so far are fantastic! Have you looked into batch processing? Batch processing is a process that lets you run a set of filter techniques across a bunch of images at once. If it's possible, you could develop a handful of techniques, and batch process all the sprites and textures using each technique. Then, most likely, one of your techniques will work great on one sprite, and another will work good on a certain texture, etc. Once the best is found for each sprite/texture, artists can touch up a sprite at a time.

Anyway, wow, these look better than most, if not all the other upsize projects I've seen!

@elend : I'm reading :) I'm also trying new solutions when they pop up (artomatix and some others) and I'm dying that real life isn't giving me the right conditions to continue this as much as I'd love to.
(to my taste) I haven't found really better than mixing both Nvidia 8x upscaling systems (normal and photo enhancer), then downscaling them with nearest neighbour setting. This is still not perfect, but that's my favourite.

Also, the fact that the HQ pack isn't working ad expected (textures are not rescaled down) has lowered my motivation : I first need to understand how to make it work, I need to get all fixed sprites projects, learn how to make a new wad using them.
All this on top of just doing/cleaning the upscaled versions :)

Since you have an account, can't you run everything through the Nvidia thing and dump the result in a file sharing service? Because others cannot register as you've read.

Well, @hidfan first of all; Good to see that you are still working on it. I know that this stupid reallife can be a real bummer sometimes.

In this regard, why not think about „distributing“ some work? I‘d love to help with cleanup. I think I can do this. Others in this forum would probably make a wad out of it, etc. What I‘m trying to say is: I love your nvidia upscale and also want this to happen. :D

All my sources should be available on my google drive (first post link?).

I'll see if i can batch quickly the other things I already prepared.

On 10/17/2018 at 3:09 PM, hidfan said:

(to my taste) I haven't found really better than mixing both Nvidia 8x upscaling systems (normal and photo enhancer), then downscaling them with nearest neighbour setting. This is still not perfect, but that's my favourite.

I suspected that the images you posted are not simply a result of processing with NVIDIA's neural network and have been post-processed. Can you describe in detail what you did and with what tools?

In the meantime, I realised that so far I've been overlooking a setting in waifu2x called TTA (no idea what that means actually). By default it is set to 1 but higher settings supposedly yield better quality upscales at the cost of longer processing times.

This is what I got by resizing the sprites 4x with TTA set to 4, then converting the image to the Doom palette with GIMP (no dithering) and scaling down to 2x with nearest neighbour:

This is zoomed up 2x using nearest neighbour for better visibility, original sized image below:

The sprites were originally against a solid grey background. I applied Gaussian blur at 2 pixel radius, then pasted the original unblurred sprites over this, to protect their edges.

Looks very nie! Let's do the whole set (using smooth doom as base). ;)

In this directory, You can find the upscaled files from
-Nvidia (8x) standard technique and "PhotoHallucination"" enhanced" (b suffix)
-Topaz' AI Gigapixel (6x)

Alpha masks are not uspcaled, I've yet to find a tool that can do it properly, for now they do almost nothing.