As of C++20 you'd better write your own Unicode library. In fact, in any version of C and C++ so far you'd better write your own Unicode library since there is pretty much zero support in all standards.
So for Windows you convert from UTF-8 to UTF-16, then copy result into NUL-terminated sequence of wchar_t and then feed it to the WinAPI you need.
Sounds like a good solution. Going from utf8 to utf16/32 is straightforward. Of course you need custom code for that, but it's pretty trivial. And since apparently wide characters actually exist and I didn't imagine them, that's great.
That is not full support because NTFS and WinAPI allow unpaired surrogates in filenames which are ill-formed Unicode.
What is this magick of which you speak?
But I'd say this doesn't matter for games because you can't even print ill-formed string.
No idea. If you can display Unicode text in your game (stb, SDL, Freetype, etc.) then you can display Unicode text in your game.