Investigating neural network upscaling for STK textures

[Calinou]

Hi,

I've recently toyed with ESRGAN and SFTGAN to upscale textures in old games. This is how I ended up releasing a neural upscaled texture pack for Quake 2 a few months ago

I've gotten pretty decent results above by also using a denoiser on the output – I used GIMP's noise reduction filter for this. It tends to look better for cartoony textures overall.

Some textures in SuperTuxKart (but not all) tend to be a bit blurry, especially on higher resolution displays. Would the developers be interested in me providing upscaled variants for those? I've noticed a few in specific tracks already, but having a more complete list would be even better. Of course, we need to balance this with the distribution size, but we can probably get substantial improvements in graphics quality with a moderate increase in download size.

[tempAnon093]

[I'm not a dev]
Can't hurt to try. If they look better, then I see no reason we wouldn't add them!

It's worth mentioning the Game Upscale Discord room (if you aren't already a member; I didn't check), there are some ERSGAN models which may work well for cartoony stuff like DigitalFrames.

[drummyfish]

This is a great topic.

I did some pixel art upscaling in a similar way, just didn't use neural nets but classical pixel art upsaling algorithms found in emulators etc. That created a good initial version which I then manually adjusted in GIMP. (BTW if someone wanted to use machine learning to create algorithms for a better automatic pixel art upscaling, I'd be very interested.)



I think fore the best quality you always have to manually edit the result of automatic upsace, even that generated by a good neural net. The amount of manual work will of course be much smaller than complete upscale by hand.

With machine learning such as the programs you used I see a few potential issues and dangers with copyright -- I don't know if that's the case, but I suppose that if a non-public domain dataset is used to train the net, then the images it will generate may end up being a derivative work of a copyrighted work, i.e. the images in the dataset, so they may be either non-free, or may be burdened by the dataset license. This probably depends on each specific case, but I think the danger is there. Does anyone know anything about this?

Also compiling and using all the these machine learning tools is not very easy, I remember having had hard time setting up a style transfer program and in the end was only able to use the SW (non-GPU) implementation, because I didn't want to use any proprietary drivers and technologies, so the program was slow and unusable. There are web tools but they insert watermarks and have various ugly conditions on what you can do with the results etc. Does any simple, good, offline, fast and free as in freedom tool exist for this? With such tools more artist would definitely do this.

[Calinou]

Update: I've applied Waifu2x + SFTGAN for a 2× upscaling factor on STK's textures. It can give pretty good results sometimes: https://imgsli.com/MjI2MDA

However, I think it's better to perform an upscaling pass on a case-by-case basis for a few reasons:

• The main textures that suffer from low texture density are road textures and terrain textures. The rest is usually fine.
• We don't want to increase the download size too much, since many people still have slow ADSL connections (or worse).
• We don't want to increase loading times and GPU memory requirements too much as well.

If you want to test the neural upscale pack I've made, use this link: https://0x0.st/ilZB.zip (expires around 2020-10-31)
To use it, extract the ZIP, make a backup of your data/textures folder, replace all textures in the data/textures folder, then restart the game.
Keep in mind this is an early version and many textures will appear to be broken (specifically those with an alpha channel, since I screwed up something in the upscaling script).

If you'd like, I can contribute upscaled textures for select tracks

[QwertyChouskie]

Wow, the textures it upscaled look a lot better! (Case in point, grass2.jpg) That being said, it looks like it didn't do all of them (e.g. stk_generic_dirtRoad_a.png).

For highly visible textures (grass, roads, etc.) this seems like a great thing! Obviously, not all textures need upscaling, but for the ones that do, its awesome!

The Android release ships reduced-res textures anyways, so increasing size isn't a concern there, and for the desktop release, even 100-200MB more at worst wouldn't be noticed by hardly anyone (given that triple-A titles can weigh in at 100s of GBs nowadays).

[QwertyChouskie]

After a second look, it looks like stk_generic_dirtRoad_a.png is just a stretched copy of roadDirt.jpg. I exported a copy of the upscaled roadDirt.jpg and saved it as stk_generic_dirtRoad_a.png, and man, what a difference!

I'd love to see some of the track-specific texture upscaled too, especially the ones used in Black Forest.

I actually wonder what the highly visible terrain textures (roadDirt, grass2, etc.) would look like upscaled 4x, it may be worth the VRAM usage for these highly-visible textures if they look much better scaled to 4x.