OpenSpades Goxel support

[bzt]

Hi,

I've seen it here that there are issues with saving Voxlap format in Goxel, so I was thinking I could help, and took the liberty to check it out.

Bad news is: no fix possible sadly.

1. Terravox (and according the only spec I could find) the dimensions in a Voxlap file are fixed at 512 x 512 x 64
2. Any other dimension won't work (but unfortunately they exists, because many editors allow bigger voxel images)
3. The encoding format in which Goxel saves isn't the same as described in the spec. According to the spec, there should be 8 bytes packets, however it looks Goxel stores different packets.

Since there's no magic bytes in the Voxlap format, and dimensions aren't stored in a header either, plus Goxel's developer hasn't responded for over 5 years, I'd suggest to forget about Goxel, and add support into OpenSpades for another format instead (maybe .kv6 or .vox which are already supported by Goxel properly).

Your chances are better that OpenSpades devs will accept a PR than guillaumechereau fixing the exporter for a broken by design format according to a 5 years old and still open issue.

Hope this helps,
bzt

[Julius]

.kv6 works in Goxel? That would at least allow making new weapons and player-models I think. Edit: it can import .kv6 fine, but not export it.

Might be worth investigating what other map formats are used in AoS/OP. Maybe this odd format is only used for the procedurally generated hightmap maps?

[Julius]

Hmm, did you see this: https://github.com/xtreme8000/libvxl/

Apparently that is what Better Spades uses.

Edit: and also already ported to Goxel: https://github.com/xtreme8000/goxel

So I guess level editing is covered with that. Just needs .kv6 export functionality.

[bzt]

.kv6 works in Goxel? That would at least allow making new weapons and player-models I think. Edit: it can import .kv6 fine, but not export it.

Hmmm, you're right that's import only indeed. It was mentioned in the issue, so I've thought it should be supported. Here's a 4 years old issue about kv6 export, still unsolved and still open...

Might be worth investigating what other map formats are used in AoS/OP. Maybe this odd format is only used for the procedurally generated hightmap maps?

It looks like there's import/export support for kvx in Goxel, whatever that format might be. It's implemented in the same file as kv6 so I suppose it is kv6' successor? Anyway, also has magic bytes and a header with dimensions, so should be perfect for the job. (But again, I don't know if Goxel's kvx support is buggy or not, I haven't used it myself, I've just read the source code)

Hmm, did you see this: https://github.com/xtreme8000/libvxl/

I think the example C code in the AoS spec is simpler to use and easier to adapt to Goxel. But good luck with either anyway! Honestly I don't think guillaumechereau will merge any of that, considering he is giving the silent treatment to you for 5 years now...

Why don't you just use Terravox may I ask? It is cross-platform and FOSS, and was designed for the purpose of creating and managing VXL files specifically. Seems like the best option to me.

Cheers,
bzt

[Julius]

The Goxel fork of the BetterSpades developer with .vxl support will work I guess. Maybe they will also add .kv6 export. I made a feature request for it.

Terravox is just a high map terrain editor. It can be used as part of the mapping pipeline, but you can not make real detained maps with it. Also, 7 year old abandoned code-base does not instill much confidence

[bzt]

The Goxel fork of the BetterSpades developer with .vxl support will work I guess.

Maybe, but AFAIK libvxl saves arbitrary sized volumes (see libvxl_create gets the bounding box differences as arguments). If OpenSpades takes AoS compatibility seriously, and expects dimensions to be 512x512x64, just like Terravox does, that could be a problem.

(BTW libvxl does a neat trick on load: it parses the vxl twice. First, it loops through the chunks and measures the dimensions, then it loops through again, this time filling up the mesh. Maybe OpenSpades could use the same trick to load arbitrary sized maps too?)

On a different note: I think I've figured out why the Goxel vxl export is buggy. It looks like the write_map function is almost exactly the same as in the AoS spec. However there are differences in the "is_surface" function, and I believe when the mesh is copied into the map array at line 254, the coordinates are seriously messed up.

{l Code}: {l Select All Code}

        pos[0] = 256 - x;
        pos[1] = y - 256;
        pos[2] = 31 - z;


This has to be the reason for the strange and incompatible RLE runs in the output (packet heights and lengths encoded as 00 40 3f 0f when that should be 00 00 00 00).

Maybe they will also add .kv6 export. I made a feature request for it.

I might be able to help with that (assuming xtreme8000 won't put that on hold for 4-5 years like guillaumechereau does).
FYI, there's already a kv6 export implemented in libvxl, but it is commented out for some reason.

Terravox is just a high map terrain editor. It can be used as part of the mapping pipeline, but you can not make real detained maps with it. Also, 7 year old abandoned code-base does not instill much confidence

For the latter, are you sure it's abandoned and not just finished? TBH vxl is an old format, which hasn't changed over the years the slightest bit, and if Terravox is considered feature-full and bug-free, then I see no reason why it should need to receive any new commits.

As for the former, if you say it's not good enough to create real detained maps, that's a perfectly valid reason to look for alternatives.

Cheers,
bzt

[Julius]

For now this seems to do the trick for .kv6 import/export: https://github.com/simlu/voxelshop but of course having everything in Goxel would be probably nicer. If you can make a PR to the BetterSpades Goxel fork for enabling .kv6 export that would be great though. Voxelshop seems to be able to import .kvx bu otherwise it is more focused on other voxel formats (and the Java GUI is by far not as nice as Goxel).

Do you think it is worth trying to fix the exisiting .vxl support in Goxel? I assume the libxvl integration would be more advanced/better anyways?

There is an open issue on Terravox to import and place .kvx map-models (buildings etc.), which would be already a big step. But caves and so on would still not be possible. So I would not call it finished, but I guess it does indeed function sufficiently well for a narrow use-case.

[bzt]

and the Java GUI is by far not as nice as Goxel

Hard to argue about that :-)

Do you think it is worth trying to fix the exisiting .vxl support in Goxel?

Well, I've tried.

First, I was right, copying the mesh into the map array was totally bad. I've fixed that perfectly.
Second, the example code in AoS spec (the one that was just blindly copy'n'pasted into Goxel) is buggy as hell. I've tried to fix that too, it is almost perfect (not entirely, generates some litter at the end of the file). I've attached the patch to the original Goxel issue, you can take a look.

My biggest WTF BTW, that Goxel can only be compiled with debug, asan, bound checks and all the sanity checker support on Linux, and the resulting executable is SOOOOOO F*ING SLOOOW, that's unbelievable. It takes about 3 minutes for the popup menu to be displayed after I click on "File" (not joking, several minutes!!!). I've tried to run "scons mode=release" to turn all that overhead off, but that one crashes during build and won't compile... Arghhhh. (Quick and dirty workaround: delete src/formats/gltf.c, and after that you'll be able to compile in release mode. Still incredibly slow though, but not insanely slow at least.)

I assume the libxvl integration would be more advanced/better anyways?

Definitely, the only issue is, as I've said the libvxl saved vxl isn't compatible with the AoS version of vxl.

BTW, I've just found Voxie, another FOSS voxel editor, which supports vxl (however i'm not entirely sure that this vxl is the same as that vxl... no example voxel images, so I can't tell for sure. But it has Python import/export plugins, so should be simple to add new formats.)

Cheers,
bzt

[Julius]

Thanks for looking into this, hopefully this motivates the main developer to fix it. btw: it seems like Goxel requires dual-licening of contributions because their mobile version is closed-source. Maybe that is why your pull requests are not accepted?

I have a bit of a hard time believing that the BetterSpades developer is making something that only works on BetterSpades, given that it is supposed to be backwards compatible with AoS. But who knows, it is not impossible. It is on my back-log of things to test with OpenSpades, but right now I was busy looking into the player models and weapon replacements which are higher priority.

Obviously if someone was really motivated in fixing up Terravox to allow map-model import and real sculpting of terrain, that would be maybe the perfect solution. Making .kvx map-models should already work in both Goxel and Voxelshop.

Edit: I'll look into Voxie, but it also seems to be abandoned and not really that functional UI wise to to use as a map-editor (similar to Voxelshop).

[bzt]

Thanks for looking into this, hopefully this motivates the main developer to fix it.

Let's hope so! Even if I get a flawless fix, it's no good if he doesn't merge it.

btw: it seems like Goxel requires dual-licening of contributions because their mobile version is closed-source. Maybe that is why your pull requests are not accepted?

You're welcome. About my PR, I don't think so, I've signed what he has requested. I honestly don't know why he is refusing, my PR just adds one more source file to the repo, it changes nothing, breaks nothing and has no conflicts.

I have a bit of a hard time believing that the BetterSpades developer is making something that only works on BetterSpades, given that it is supposed to be backwards compatible with AoS. But who knows, it is not impossible. It is on my back-log of things to test with OpenSpades, but right now I was busy looking into the player models and weapon replacements which are higher priority.

As I've said, BetterSpades (and libvxl) does more than original AoS, can handle any size. This means vxl are saved with arbitrary dimensions, but also means original AoS vxl files with fixed dimensions will be loaded no probs too, so it is backward compatible.

In order to load libvxl saved maps into OpenSpades, either it should use libvxl too, or you must make sure of it, there's a voxel at minimum and maximum positions, therefore bounding box returns 512 x 512 x 64 (and in that case libvxl saves a vxl that's AoS compatible and surely can be loaded in OpenSpades with the current loader).

Obviously if someone was really motivated in fixing up Terravox to allow map-model import and real sculpting of terrain, that would be maybe the perfect solution. Making .kvx map-models should already work in both Goxel and Voxelshop.

Yeah, that's why I think it would worth adding kvx loading in OpenSpades. (As a bonus, there would be no need for kv6 either, as models can use kvx too. You only have to convert the existing assets once (.kv6/.vxl loading and .kvx saving works in Goxel), and all future development could use Goxel and a single format afterwards)

Edit: I'll look into Voxie, but it also seems to be abandoned and not really that functional UI wise to to use as a map-editor (similar to Voxelshop).

That's a pity, it looked good at first.

What's with this FOSS voxel editor scene? It looks like there's no usable one (either they are abandoned, missing export formats, slow as a pregnant snail or proprietary). Maybe when I've finished with the TirNanoG Player, I'll work on a small, easy to use FOSS voxel editor that supports file format plugins. It looks like there's really a need for that.

Cheers,
bzt

[Julius]

So generating a terrain in Terravox with the right dimensions and then further mapping in the libvxl Goxel version should work then as long as you don't exceed the original size?

I think there are quite some Voxel editors, but no or few that lend themselves to voxel map editing as most voxel games are generating maps procedurally with a focus on altering the maps in game directly. OpenSpades is really the exception here as it follows a more traditional map making paradigm and the tools for autogenerating maps are underdeveloped for it as well.

[bzt]

So generating a terrain in Terravox with the right dimensions and then further mapping in the libvxl Goxel version should work then as long as you don't exceed the original size?

Almost :-) Terravox uses fixed dimensions, so you can only use right dimensions with it. With libvxl, it's not just it shouldn't exceed, it cannot be smaller either. The dimensions must be 512 x 512 x 64 exactly to make it compatible.

Why am I thinking this? If you look at the AoS spec examples, both outter loops have the same hardcoded size. Look:

{l Code}: {l Select All Code}

//load_map
   for (y=0; y < 512; ++y) {
      for (x=0; x < 512; ++x) {


and

{l Code}: {l Select All Code}

//write_map
   for (j=0; j < 512; ++j) {
      for (i=0; i < 512; ++i) {


Similarly in Terravox's vxl.cpp we can see the same hardcoded values:

{l Code}: {l Select All Code}

//vxl::load
        for (int y = 0; y < 512; ++y) {
            for (int x = 0; x < 512; ++x) {


and

{l Code}: {l Select All Code}

QScopedPointer<Terrain> terrain(new Terrain(QSize(512, 512)));


On the other hand, there are no such thing in libvxl at all, everything is parameterized (as a matter of fact, the number "512" does not even appear in libvxl):

{l Code}: {l Select All Code}

bool libvxl_create(struct libvxl_map* map, size_t w, size_t h, size_t d,
               const void* data, size_t len) {


and

{l Code}: {l Select All Code}

//libvxl_write
   for(uint32_t y = 0; y < map->height; y++)
      for(uint32_t x = 0; x < map->width; x++)


It also has a libvxl_size function which guesses the dimensions by iterating through the packets (here x, y must be the same, returned in *size, and z returned in *depth).

So libvxl is a much better and more universal solution, but because of that it's not compatible with AoS/Terravox, except when map->width and map->height is 512 (as for loading, AoS/Terravox is just one case, so original assets can be loaded with libvxl no probs.)

I think there are quite some Voxel editors, but no or few that lend themselves to voxel map editing as most voxel games are generating maps procedurally with a focus on altering the maps in game directly. OpenSpades is really the exception here as it follows a more traditional map making paradigm and the tools for autogenerating maps are underdeveloped for it as well.

I see. But it's not just about maps; there's the issue of missing kv6 export, the buggy vxl export as well as the outrageous performance of Goxel. Anyway, what I'm thinking is a small voxel editor base, which would be expandable with Lua plugins. You could simply write tools to autogenerate different height-maps for example with Lua scripting if you want to use the editor for maps. Would you be interested in something like that?

Cheers,
bzt

[Julius]

Personally I am not even that much into voxel games, but OpenSpades seems like a nice "minimalist FPS" kind of game that is relatively fun to play and should in theory at least have an low entry barrier for modding contributions. Hence why I am mostly interested in proper tool-chains to enable the latter.

[bzt]

Hi Julius,

Good news, I've submitted a patch to https://github.com/xtreme8000/goxel/ which adds "vxl (AoS, OpenSpades)" option to the export menu. This initializes the libvxl library with fixed dimensions, so it saves OpenSpades compatible maps for sure (import works with arbitrary sizes, that part did not need any modification).

But it's not all happy and laughter, I've loaded a Terravox map and saved with this libvxl Goxel version, and although the file size is the same this time, but it looks like there's something not quite right with the alpha channel. I don't know if this matters or not. I've added both files in the attachment, could you please check if it works okay with OpenSpades?

Could it be that libvxl does not support alpha channel (and is it supported / needed in OpenSpades in the first place)?

Cheers,
bzt

[Julius]

Hmm, I am not even sure how to start a custom map without running a server. Something I need to figure out I guess, or alternatively run the Piqueserver on localhost. Give me a few days as I am currently busy with something else.

But thanks for working on this!

[Julius]

Maybe your alpha issue is related: https://github.com/guillaumechereau/goxel/issues/238

[bzt]

You're welcome! No hurry, I'm working on other projects as well. Let me know if/when you need anything else!

About the alpha issue, I don't know. I haven't touched the original Goxel vxl loader, nor the libvxl version, just the exporter part. For the original Goxel vxl exporter, I've only changed the coordinates and the dimension, but it is true that colors with less than 128 alpha weren't saved (this should be fixed with my patch. but again, this is when saving the vxl maps, not when loading them).

I've checked the Goxel vxl source, but there's absolutely nothing about alpha cropping. Which is bad news, because it means there must be some kind of common post-process filter after loading voxel images, no matter the format. In this case it doesn't matter if the original vxl code or if libvxl is used in vxl.c, this has to be fixed by guillaumechereau, but since that issue is over 1 year old and unresolved, I doubt he will respond any time soon.

A simple workaround could be to set the alpha to 255 on all colors after loading, but that would make all the shadows disappear. Hmmm, okay, I've checked, and it looks like that's exactly what xtreme8000 does after successfully decoding the map using libvxl:

{l Code}: {l Select All Code}

               mesh_set_at(image->active_layer->mesh, &it,
                        (int[3]) {map_size / 2 - 1 - x,
                                y - map_size / 2,
                                map_depth / 2 - 1 - z},
                        (uint8_t[4]) {BLUE(color), GREEN(color),
                                   RED(color), 0xFF});


(Note how the last part of the color is a constant 0xFF, and also that this is not happening in the libvxl source, rather in goxel/src/formats/vxl.c).

I honestly don't know if this workaround is good. I've never used AoS nor OpenSpades, especially not with shadows in maps.

Cheers,
bzt

[Julius]

Looks like the person who forked Goxel is quite open to contributions. They already backported the libvxl stuff:

https://github.com/pegvin/goxel2

[bzt]

Looks like the person who forked Goxel is quite open to contributions. They already backported the libvxl stuff:

https://github.com/pegvin/goxel2

Wow, "open to contributions"? You weren't kidding! That repo is already more than 150 commits ahead of guillaumechereau's repo! (And 0 behind.)

But if they have already merged libvxl support, is there anything I could do for you? Let me know!

Cheers,
bzt

[bzt]

They accepted my patches, now Goxel2 not only supports OpenSpades vxl, but Minetest MTS and Minecraft NBT Schematics as well!

Furthermore, they have put together a simple, but working webpage for Goxel2.

Cheers,
bzt

[Julius]

Nice! I think this Goxel2 will be the way forward

[bzt]

Nice! I think this Goxel2 will be the way forward

Yeah, I agree! The new maintainer is nothing like @guillaumechereau, he is very easy to work with! And @pegvin also has a clear vision what has to be fixed (performance mostly, and the lack of thumbnails), and having a clear goal is very important for any project.

BTW, you have opened an issue about veloren problems, if you could write a detailed list what's wrong and what would be the expected result, I could look into that too. As far as I know, veloren uses .vox files and .vox file support is already pretty complete in Goxel / Goxel2, so I'm not sure what the issue might be. Is it a missing veloren-specific chunk or something?

(The best would be if you could provide an example .vox file which does not work along with a screenshot from veloren about how the model supposed to look like. That would help a lot locating the problem.)

Cheers,
bzt

[Julius]

Last time I checked there was a problem with .vox file layers and also the export. The link I shared on that issue has an example Veloren character .vox file. I am sadly currently low on time to look into it in detail.

[bzt]

The link I shared on that issue has an example Veloren character .vox file.

Sadly no. Your link leads to the veloren's manual, which has a "reference models" link true, but it is a google drive link which returns 404. And even if it were working, I still have to know how the correctly rendered model looks like. That's why I was asking.

I am sadly currently low on time to look into it in detail.

No hurry.

Cheers,
bzt

[Julius]

I tested some of the models from here:

https://drive.google.com/drive/folders/ ... Wb_V-w456q

It actually works much better than before, but it would be necessary to actually test the import into the game. It seems like the point of origin is not respected by the .vox exporter.