Investigating neural network upscaling for STK textures

[Calinou]

Hi,

I've recently toyed with ESRGAN and SFTGAN to upscale textures in old games. This is how I ended up releasing a neural upscaled texture pack for Quake 2 a few months ago

I've gotten pretty decent results above by also using a denoiser on the output – I used GIMP's noise reduction filter for this. It tends to look better for cartoony textures overall.

Some textures in SuperTuxKart (but not all) tend to be a bit blurry, especially on higher resolution displays. Would the developers be interested in me providing upscaled variants for those? I've noticed a few in specific tracks already, but having a more complete list would be even better. Of course, we need to balance this with the distribution size, but we can probably get substantial improvements in graphics quality with a moderate increase in download size.

[tempAnon093]

[I'm not a dev]
Can't hurt to try. If they look better, then I see no reason we wouldn't add them!

It's worth mentioning the Game Upscale Discord room (if you aren't already a member; I didn't check), there are some ERSGAN models which may work well for cartoony stuff like DigitalFrames.

[drummyfish]

This is a great topic.

I did some pixel art upscaling in a similar way, just didn't use neural nets but classical pixel art upsaling algorithms found in emulators etc. That created a good initial version which I then manually adjusted in GIMP. (BTW if someone wanted to use machine learning to create algorithms for a better automatic pixel art upscaling, I'd be very interested.)



I think fore the best quality you always have to manually edit the result of automatic upsace, even that generated by a good neural net. The amount of manual work will of course be much smaller than complete upscale by hand.

With machine learning such as the programs you used I see a few potential issues and dangers with copyright -- I don't know if that's the case, but I suppose that if a non-public domain dataset is used to train the net, then the images it will generate may end up being a derivative work of a copyrighted work, i.e. the images in the dataset, so they may be either non-free, or may be burdened by the dataset license. This probably depends on each specific case, but I think the danger is there. Does anyone know anything about this?

Also compiling and using all the these machine learning tools is not very easy, I remember having had hard time setting up a style transfer program and in the end was only able to use the SW (non-GPU) implementation, because I didn't want to use any proprietary drivers and technologies, so the program was slow and unusable. There are web tools but they insert watermarks and have various ugly conditions on what you can do with the results etc. Does any simple, good, offline, fast and free as in freedom tool exist for this? With such tools more artist would definitely do this.

[Calinou]

Update: I've applied Waifu2x + SFTGAN for a 2× upscaling factor on STK's textures. It can give pretty good results sometimes: https://imgsli.com/MjI2MDA

However, I think it's better to perform an upscaling pass on a case-by-case basis for a few reasons:

• The main textures that suffer from low texture density are road textures and terrain textures. The rest is usually fine.
• We don't want to increase the download size too much, since many people still have slow ADSL connections (or worse).
• We don't want to increase loading times and GPU memory requirements too much as well.

If you want to test the neural upscale pack I've made, use this link: https://0x0.st/ilZB.zip (expires around 2020-10-31)
To use it, extract the ZIP, make a backup of your data/textures folder, replace all textures in the data/textures folder, then restart the game.
Keep in mind this is an early version and many textures will appear to be broken (specifically those with an alpha channel, since I screwed up something in the upscaling script).

If you'd like, I can contribute upscaled textures for select tracks

[QwertyChouskie]

Wow, the textures it upscaled look a lot better! (Case in point, grass2.jpg) That being said, it looks like it didn't do all of them (e.g. stk_generic_dirtRoad_a.png).

For highly visible textures (grass, roads, etc.) this seems like a great thing! Obviously, not all textures need upscaling, but for the ones that do, its awesome!

The Android release ships reduced-res textures anyways, so increasing size isn't a concern there, and for the desktop release, even 100-200MB more at worst wouldn't be noticed by hardly anyone (given that triple-A titles can weigh in at 100s of GBs nowadays).

[QwertyChouskie]

After a second look, it looks like stk_generic_dirtRoad_a.png is just a stretched copy of roadDirt.jpg. I exported a copy of the upscaled roadDirt.jpg and saved it as stk_generic_dirtRoad_a.png, and man, what a difference!

I'd love to see some of the track-specific texture upscaled too, especially the ones used in Black Forest.

I actually wonder what the highly visible terrain textures (roadDirt, grass2, etc.) would look like upscaled 4x, it may be worth the VRAM usage for these highly-visible textures if they look much better scaled to 4x.

[tempAnon093]

I didn't get a chance to test the upscaled version, but bumping for interest. Is it worth putting the properly-working upscaled textures in the next release?

[Alayan]

Better textures don't cause compatibility issues and can drastically improve how the game looks.

So yes definitely, better looking textures are most welcome, any improvement on the frequently visible low-quality textures would be a very useful contribution.

[drummyfish]

Out of curiosity, what's are the copyright rules for AI created data? I.e. if I train an AI on a data consisting of non-free photos and the AI then synthesizes a new but similar image, who owns the rights? The programmer? The user? Is it a derivative work? Is it public domain? Because this is not clear to me I try to stay away from AI for now.

[QwertyChouskie]

IANAL, but as long as the resulting image doesn't clearly have parts of the non-Free images, you should be fine.

Or another way to think about it: When we humans create images, we've all seen lots of non-Free images in our lifetime, but as long as the image we create is not a clear copy or rip-off of one of said images, we own the copyright.

[tempAnon093]

Out of curiosity, what's are the copyright rules for AI created data?

IANAL
Reddit - Is it legal to use copyright material as training data? - (alternate links to avoid trackers: kddit or teddit)

I am planning to train a Neural Network on some videos which happen to be copyrighted. Is it illegal to use them?

"I want to preface this by saying I am not a lawyer, but I do run a company where this issue is critical to our business and have spoken with half a dozen lawyers on the subject in quite a lot of depth.
If the only restriction is copyright then it's absolutely legal so long as the original images cannot be reconstructed from whatever offering you have.
The reason for this is really very simple: copyright refers to the set of rights and restrictions around producing copies. If you're not producing a copy it doesn't apply."

If anything, it would probably qualify as fair use or not an issue.
What exactly do you mean by 'similar'? That's probably the key issue here. For example, if you trained an AI on novels and it created a new book that clearly depicts the existing character Harry Potter as the main character, it could be infringing on the copyright of the character rather than any particular book.

[tempAnon093]

I have tried emailing Calinou twice with no response, so I may look into doing the same techniques myself. However, my GPU might not have enough VRAM to properly upscale the textures. If that's the case, I will write an easy-to-follow tutorial and a simple bash script to automate.
(My only issue is that since I'm new and inexperienced, I don't want to submit something 'good enough' that no-one experienced will bother doing it properly.)

[tempAnon093]

I've done a quick test to see what's easily possible. The results are promising! (I recommend opening full screen to see the difference)
https://imgsli.com/NDkyMDM
---
Important notes:
- The quality of the original image is extremely important. Running models on stk_generic_dirtRoad_a.png (current texture, 1024x1024, blurry) was worthless, running it on road_dirt.jpg (512x512, source image that the other one is based off) gave an excellent result.
- I upscale textures using a workaround, which is still a great improvement but someone with a better GPU can probably do it slightly better. It's an easy process, especially for any devs.
- This wiki is extremely useful: https://upscale.wiki/ , they also have a useful Discord but I don't use it.
- I haven't tried transparent images yet, the wiki has tips for it.

What I did to get the above image:
1) Install ESRGAN: Install guides on the wiki. It's a Python tool.
X) Since my GPU doesn't have enough VRAM (I only have 2GB), I needed to use this set of work around scripts that splits the image into smaller tiles and patches it back together. I'd prefer someone who doesn't need this to do upscaling as it can have an impact on quality.
2) Find the best quality source image you can and put it in the correct folder for ESRGAN. Slight blur will have big impact, so really try and get the best. In this case, road_dirt.jpg was far better than stk_generic_dirtRoad_a.png.
3) Download and test models. Try a few models to get the best results, this image used "Ground" by ZaphodBreeblebox which is trained for natural ground like grass and dirt, others had good results too.

[QwertyChouskie]

For reference, here is a re-upload of the original zip of textures: https://jacobspctuneup.tk/STK/ilZB.zip