How important are libre development tools to you?

[nnesse]

I've been wondering how imporatant the use of free (as in freedom) development tools are to the average aspiring game developer here. I see many instances of projects that settle for a kind of theoretical support of free software tools but in practice a number of non-free programs are really driving the project forward. Inadiquite support of Blender is a good example. Many open source game engines import assets via COLLADA but Blender does not support exporting animation data within COLLADA (at least last time I checked). I would argue that a game that depends on animated models but is using COLLADA is making a questionable compromise. I think it's similarly problematic to use cmake to generate Visual Studio project files when MinGW is a viable option.

What I worry about is how these sorts of decisions can change the makeup of a project's contributers to the point that directions are taken that work against the free tools. There ought to be a network effect where every free software project elevates several others. So, for myself, I put the similarly strict requirements on the tools I use as the code I link with. Since so few libraries and tools put the same priority on this as I do I've had to do a lot more things myself from scratch. I've started working on a graphics/game engine with a tight integration with OpenGL, Blender, and various GNU software. It's been fun but time consuming. Since so many engine projects die out I've taken a highly modular approach to development to the point that the components can at least be used even if the project dies out (see https://github.com/nnesse). I would be interested in working with anyone who is interested in low level tech and willing to take the more difficult road of only building on software with clear libre pedigree's.

[oln]

Shouldn't you also use a website that uses free software then?

[nnesse]

I assume you're talking about github vs gitorious? I've been pushing my code to both sites just to maximize the chance of getting some feedback. I agree that by my own arguments gitorious (or somthing comparable) ought to be used for collaboration. I just created a group on gitorious for this purpose: https://gitorious.org/+libre-gamedev-tech. I'm trying to move my current gitorious projects (gltext, glbindify) to the team page but the site keeps timing out on me. Just message me if you want to join the group

Rereading my original post I've realized that my definition of what software I would like to avoid is not clear enough. The tests that I apply is that all software I would use or be interested in helping to develop are:

1) Must be FOSS itself
2) Must provide superior support to other FOSS software and otherwise not encourage the use of non-free software
3) Must be buildable using software also following #1 and #2 (apply recursively)

That is, the whole software stack behind the engine and tools can be built from scratch and used effectively without leaving the FOSS world.

[onpon4]

I only use libre software and encourage others to do the same, but it sounds like you're saying that stuff like 3-D models that were made with proprietary software should be rejected for this reason. Well, I would disagree with that conclusion. If reading it requires proprietary software, of course it should be rejected. But if it's just that someone else sacrificed their freedom to make it, then what's done is done, and there's really no point in rejecting it.

Similarly, I wouldn't reject a useful contribution to one of my programs if the contribution was typed in a proprietary text editor, regardless of whether or not the features of the proprietary editor made the work easier.

[nnesse]

What I'm getting at is that the infrastructure that I depend on should not have a propritary bias. The original source of the contribution doesn't matter so long as it can be conditioned into a form where it fully ineroperates with the FOSS tech. Going back to the Blender case, If I accept a model with animations that cannot be imported into Blender then my project is now dependant on non-free modeler from a development process standpoint because only people who can afford and are willing to run such programs can improve or change that animation. The COLLADA format was supposed to provide the neccisary interoperability but in practice it rarely works as intended. Every tool has their own interpretation of the standard so when COLLADA is used you are effectively adopting a dependency on the program used to create the model. It's a similar story with cmake's ability to generate MinGW makefiles on windows. If user's are invited to generate Visual Studio solutions then the MinGW toolchain approach will only work intermittantly if ever. I understand how tempting making these kind of compromises is which is exactly why I would like to make some tech that improves the pure FOSS development experience.

[asdv]

For me a lot off creative FOSS software just lacks the usability I need to spend my free time using them. For example having recently had to try and compile something (SDLgfx) with both Visual Studio and MinGW I personally can’t see MinGW as a viable option. The amount of pain required to get everything set up and not-working was just too much.
And God help you if you want to compile for 64-bit. In Visual Studio it was as easy as choosing x64 on a drop-down list and I was able to find that option without consulting a manual or even googling since things are somewhat logically laid out.
Since you’re already on Windows I can’t see it mattering if you have to install VS.

And I don’t think I’ll ever be able to replace the adobe suite. Just a simple thing like I can’t find a single FOSS paint program that lets me do something as basic as move/copy multiple layers blows my mind.
GIMP is a perfect example; the fact that it took them over a decade of people complaining before they implemented a single window mode speaks volumes of what they think of their (potential) users. GIMP in general also lacks any discoverability whatsoever and as long as they spend years fiddling around with GEGL instead of writing a coherent design document I don’t see that changing.
I don’t want it to sound like I’m coming down too hard on FOSS, I use Linux most of the time myself, but the tool chain needs a lot of improvement.

[nnesse]

I agree there are many projects that need a lot of work. That said the community should be looking for the best emerging solutions and rally around them. A lot of people have understandably bad memories of working with MinGW , buggy, terrible web presence, etc, but now we have MinGW-64 (http://sourceforge.net/projects/mingw-w64/) which I have personallly had good experiences with and can reccomend. In terms of getting things built with MinGW there are two projects worth looking at. MXE (http://mxe.cc/) which is a Linux makefile system which batch cross compiles a long list of popular software and libraries. If you want to work on windows take a look at the MSYS2 project (http://sourceforge.net/projects/msys2/) which is a Cygwin/MSYS replacment with good support and a real package manager which includes quite a few MinGW packages. Notibly, SDL2_gfx is built by MSYS2 and MXE so you'll probably find a solution to your build problem in one of those places.

There definitely are deficienies in the toolchain but there is also a social problem. There are some high profile projects that have a history of simply not accepting MinGW compatibility patches, the primary maintainers seem to only care about business use cases. What I want to do is get in touch with other developers who are interested in eliminating some of the biggest pain points and make new software for game developers that feels more like it belongs to the FOSS community. I can't really comment too much on the art side as I am mainly a programmer. My primary concern is the extent to which propritary tools have become institutionalized to a point where a free tool user cannot effectively contribute because there are no formats supported by both tools that preserve all the information. I don't know if the same kinds of issues exist between Photoshop and GIMP as there are between 3D modelers.

My next project, which is going to take awhile (and probably require collaborators) is a real-time renderer that operates as directly as possible on blender data. My approach will be to make a blender plugin that pickle's (http://www.picklingtools.com/) a subset of blenders RNA data then write a C++ library that loads the pickled data and creates corresponing GLSL shaders and provides an interface to control animations. The renderer will be a pluggable system mostly concerned with marshalling the data and doing the animation math. This way anyone who wants to can use the project to take different approaches to the actual rendering. Anyone who wants to follow the development or help can find it on gitorious: https://gitorious.org/blender-to-glsl

[gouessej]

Hi

My next project, which is going to take awhile (and probably require collaborators) is a real-time renderer that operates as directly as possible on blender data.

In my humble opinion, it's a bad idea as the Blender format isn't an interchange format, it's the native format of this modeler and nothing else. I think that it's the job of the modelers to provide a correct support of interchange formats. I like Blender but I have had the same troubles with animations and since its version 2.50, most importers for Quake formats have been dropped. When I succeed in updating the import scripts, the Collada export only gives me the very first frame Now I use the importers of my engine to manage those files and I export all key frames in a single OBJ file.

I understand your position. I try to stay far from proprietary softwares. The few native libraries I use are compiled without Visual Studio. I'd like to build OpenJDK without it too. I agree with you about the network effect. I often talk about the convergence of the struggles. We are stronger together. If all free softwares developers really wanted to eradicate proprietary softwares from their projects, it would increase the solidarity between them. Instead of that, I see lots of people who only care about their own things. For example, the homepage of the Neo900 shows a link to a Twitter account and a Youtube account whereas Identi.ca and GNU Mediagoblin are worth a try. I think that it's a question of priority and motivations. We shouldn't assume that all free softwares developers work on free softwares for ideological reasons.

Finally, I remind you that the preservation and the development of the alternatives are time consuming. Lots of people just want to use something convenient. I want to regain control on my data. I'm interested in self hosting and net neutrality too. I think that someone who isn't really convinced that free softwares are more "fair" would do lots of efforts to show that using only free softwares is a viable path. Actually, if you really want to be sure to stay away from proprietary softwares, you should put your own Git repositories onto your own server.

Edit.: I still have to go away from a few proprietary craps...

[nnesse]

I have made some adjustments to my approach to my blender project that might address your concern. The goal was never to copy the blender format completely. I was just going to filter it, removing objects and properties that a real-time graphics engine would not need. After exporing this idea further I found that there was more data in the format that I didn't need than I did. Also many of the fields that were potentially useful were not adaquitely documented.

So now I'm going with creating a more conventional exporter but I still try to match the blend file's data structure so that tools and blender addons I create later can cater to techichal artists without fighting the format. The refactored project is here: https://gitorious.org/blender-asset-pipe/blender-to-lua. The output format is documented in the first part of the export script. Since blenders material system's has poor documentation and naturally has many properties centered around raytraced rendering I am only focusing on exporting animation and geometric data for now. At a later point I will either create a material editor that runs outside of blender or use the PyNodes API (used by cycles but currently undocumented) to create a node based material editor in blender tailored for real-time rendering.

I have some profressional experience working with people from the AAA gaming world and one of the things that I learned is that game studios usually avoid getting dependent on file formats like Collada, they will integrate as tightly as possible with their artists tools, even creating custom GUI's within them so that the tools will always produce content the engine can consume. I want to provide a solutions like that but tailored around the best available free tools instead.

I prefer self-hosting for the same reasons but in the early days of a project like this I want to stay focused on the work and not maintaining a VPS or home server. There is a tipping point where I would setup all the neccisary SCM and collaboration tools on my own server but I'm not there yet.

[nnesse]

I moved this work to notabug.org some time back since gitorious went down: https://notabug.org/nnesse/. I've also mirrored it at https://github.com/nnesse for extra visibility.

Since my last post I've improved blender-to-lua quite a bit. I also created a new project called b2l-tools that includes a material editor tool that is capable or previewing B2L files with animation playback and custom shaders. The material editor parses the GLSL of the user specified shaders and maps the inputs to widgets so you can experiment with the shading in real-time. It can also save out the shader parameters to file to make it easier to replicate the rendering in a separate game or engine. It's a very rough around the edges tool but perhaps someone here could still make use of it.

The next steps for me are going to be to broaden the scope of the material editor to include scene viewing/editing and isolate the rendering code to allow it to evolve into a proper graphics engine.

[Vandar]

For me it's not important if a tool is open source. Freeware is welcome if it does the job, also closed source tools from the shelf, which cost money.

Important is, that the benefit is greater than the price, and that they will stay with me for a while. E.g. I don't like rental models, where I have to pay a tool freshly each month or so.

On the other hand, my current tool set only contains open source tools, but that is just by chance.

[nnesse]

There is a problem with that reasoning however. By the nature of their business proprietary software vendors not selling subscription software will tend to insert non-backwards compatible changes over time, making it amount to a subscription model either way.

There are also practical problems with depending on too many proprietary gamedev tools. It is often not possible to share data and algorithms between your code and your tools as freely as might be possible in an open source toolchain. There may be algorithms and data best applied in your game that the tool will only give you the option to "bake" to a much larger and less flexible data set. Likewise there may be transformations you are doing at run-time that would be best applied ahead of time in the tools.

A good example of this is a node based terrain generator I used one time. I got to a point where I realized what I wanted was for the terrains to be generated at run-time using seed data that changed based on what kind of challenge the user wanted. The tool neither provided the algorithms as a library or documented any of the formulas so the most I could do was to pre-generate a large set of possible maps which would have both used up lots of space and still not offered the same flexibility as my preferred approach. I don't think the lack of documentation of the formulas was an oversight. I suspect the developers presumed that having the math obscured in the compiled code was making it more difficult for others to create a competing tool.

[Vandar]

Likewise there may be transformations you are doing at run-time that would be best applied ahead of time in the tools.

In this case, you can also extract the transformation into a converter, which is run once in the release or setup process.

The other points might be valid, but they were no problems for me so far. And I must admit, even that I use open source tools, my will to change them is small. E.g. I'm using paint.net, it's my favorite graphics tool, and I really don't want to install a development environment for that. So in effect, it's the same for me as a closed source tool of equal capability. The reasons why I am using it, is that it costs nothing and it's small, easy to use and does what I need. Open source was not a prime reason for me to to chose paint.net over other tools.

If a tool doesn't do what I need, e.g. your landscape generator, I seldom try to change the tool, but I look for something that fits better.

Finally, I can understand that the landscape generator developers want to keep the formulas secret. If you want to make money with a product, you'd be stupid to publish such essential information - any competitor in the market would make use of it and you'll lose the advantage that you have.

That's different in the free/open-source world, but for companies which make money by producing and selling software, it's the right thing to do, I think.

[nnesse]

Well in the case of the terrain generator it did do what I wanted, I just wanted to be able to replicate it at run time rather than export it. It's not so much about modifying the program as it is simply having visibility into it. Another example of this problem was with a contract I did awhile back where a long sequence of images had an effect applied to them. Without the effect the images compressed excellently so it would have been much better option to apply the effect in a shader and compress the originals. Regarding doing transformations ahead of time, yes you definitely can rely on external converters etc but in a complex project the fewer steps between a content creation and run-time the fewer mistakes that will be made and the faster people can work.

What I'm talking about I think applies more to 3D game development where the technical complexity tends to be higher. Medium to large players in this field will often have a support contract with all of their 3'rd party software they use so even if they don't have access to source they can always pay to get a bug fixed or a change made in the time frame they need. For an open source 3D project this is sort of a double blow: the cost of making the tools actually needed to be productive in open source may be too high, and any proprietary tools you do use can't be used as effectively as commercial players can.

[Vandar]

Well, there is Blender, and it has a impressive even if not always easy history. I never could wrap my head around Blender, but I see impressive things done with it. These days I ponder about installing Blender again, and try to learn it - third time, this would be. Actually I'd only need a mesh modeller with PovRay export, but it's difficult to find a good one.

[Julius]

Uhm, when it comes to working on models propitiatory mesh formats like .max are a real headache, and sadly the exporter and importers for exchange formats like collada of fbx leave much to desire. In the end (except for the most basic mesh exchanges) forget about collaborating on a mesh if you aren't using the same software (and more or less even the same version of that software).
So for most purposes this leaves only Blender, at it is the only program that everyone has and that runs on everyones computer. Using .blend is also pretty much the only way you can store "source files" for your meshes in a way that someone else down the road even has a chance to ever continue where you left off; although sometimes that might mean that you have to go into the extensive version archive of Blender and download an older one. Luckily you can easily have multiple different versions installed on the same PC which makes "upgrading" a file usually possible.

The only other FOSS modeling tool that is worth a try (as it is much easier and quicker for organic modeling) is Wings3D. But I would suggest you set it to Blender3D's camera control from the beginning as at some point you will have to use Blender at least for some file converting, and being used to the controls prevents a lot of frustration.

@Vandar: Wings3D has a pov-ray exporter no?

[Vandar]

I didn't know about Wings3D. It looks very interesting to me, I'll try it later the day. Thanks for the suggestion!

Edit: Installed and gave it a quick try. Seems to be exactly what I need. Povray export is good, it even seems to have a povray integration for previews. So it seems to fit perfectly into my toolchain.

Thanks again! Saves me from the horrible Blender UI.

[Jastiv]

I'm not sure about blender and 3D modeling as I have not done much of that. but I do know I have always used GIMP for almost all my needs. I learned on that back when I began to do pixel art. Before that I had MS paint and I never took professional level graphics (ie Photoshop) classes at school. I did take autocad though, and i thought omg, why does a disk with some bites on it cost so much when the same disk with a game on it would cost like 1/15th of that. (shows how old I am I guess) Basically, I installed Linux, got the free GIMP program, saw it was a massive upgrade to ms paint and got seriously interested in making art on the computer.
Actually, free software tools are why I got into free software development, I probably wouldn't have bothered with it if it was all just proprietary tools.

[Wuzzy]

I think it's very important to insist on free software tools. Otherwise it would kind of miss the point of free software, right? It's about being consistent.

It's not good if a software that itsef wants to be free requires contributors to install any non-free tool. While I do not think any project should dictate their contributors what tools to use, contributions via free software should always be possible and generally be the preferred method. If a proprietary tool is preferred, recommended or even the only way to contribute, that should be a red flag.

This also applies to the choice of data formats. It's a bad design choice for a free software project to take some obscure file format that only one of a few proprietary programs know, unless, of course, the project is about reverse-engineering that file format . Another thing I think is critically important is to decide on the tools and file formats to use as early as possible and do it right the first time. If you add a hard dependency on proprietary early, and don't do anything about it fast, it will be harder and harder to get rid of the nasty dependency as the project moves forward.
This also has the neat side-effect that if your project has survived long enough with only free tools, it will be harder to justify the introduction of proprietary things in the workflow in general.

I am also aware that free software tools might not always be the very best in terms of quality or usability.
Too many devs take the lazy route and decide to sacrifice freedom on the altar of convenience, that's sad.
I'd argue that even if a project might not be the best one now, that should hopefully change in future as more and more people join the free software movement.
You can still indirectly support the improvement of these tools by simply using them and giving feedback, it also might increase their popularity a tiny bit. Especially if you write a lot of said software in various support forums. And again: Consistency! It would be only half as convincing if you require some non-free software in the workflow. Then people can rightfully claim that you didn't really care about freedom after all.
But, oh well, nobody said that creating high-quality free software is going to be easy, and the path of least resistance might not always be the best one.

From a purely strategic perspective, depending on something proprietary also has the risk in that the owner of the proprietary thing can royally screw you over in the blink of an eye, whether it be a malicious update or by giving a new unfair license which you are bullied to obey, or by simply rising their monopoly prices to an unpayable level. For a case study, look up the history of BitKeeper and Git.

[Jastiv]

I remember that whole thing with BitKeeper and Git. Ultimately, Git was a great gift to the free software community, and I'm glad I decided to finally switch from CVS.

[Julius]

Nice Blog about libre graphics tools:
http://libregraphicsworld.org/