Unix philosophy in game dev

[drummyfish]

I'd like to share this observation that's been bothering me.

Libre games are great because they fix one great issue of mainstream games, which is of course being proprietary. Libre games bring free SW and free culture philosophy to games. Sadly, they mostly fail to bring closely related philosophies, such as the Unix philosophy, with them.

We have to keep in mind that being proprietary corrupts in many ways -- it leads to bloat, obscurity, maximalist monolithic unmodifiable SW, a culture and a mindset of reinventing wheels being the norm, consumerism (create SW, let it die, throw it away and create another one), and so on and so forth. A great example of this is the Windows mess, the ugliest, most non elegant OS ever created. In the free SW world we don't aim to replicate this, we don't want to just add a free license, we aim to do it right thanks to being able to collaborate and share thanks to the license.

So, free SW doesn't mean just a license, it means a complete mindset of creating things in a reusable, friendly, minimalist and extendable way. The Unix philosophy tries to capture this.

My whole point is that a lot of libre game devs try to replicate the proprietary AAA games and along with them they replicate all the wrongs of the proprietary world. Too often we try to create whole monolithic games, from the ground up, with their own dedicated engines, tools, assets etc. The finished game is a little bit of an OS in itself -- it has its own GUI system, user profile system, network protocols, built in internet browser, scripting subsystem maybe even with a custom made language etc. This is wrong.

What we -- or at least a subgroup of us that is unfortunately practically non existent -- should be doing is create our games in a way in which we create our Unix tools. There is no reason to treat games differently.

For example, instead of creating a game, your whole project can be just creating a very good backend for a racing game, without any graphics or sound or IO, with minimum dependencies. Just a program that can handle racing entities. It can work as a library or just a text stream filter run from command line. With just this thing to focus on you can make it extremely good. Once this is finished, someone -- and it can be you again, but as a part of another project -- can create a rendering front end for it. There can be a 3D frontend, 2D fontend, ASCII frontend. A great frontend that can also be backend agnostic and leave a possibility of someone creating another backend. This front end is configurable and can take different asset packages to change how the cars look etc. Now you have these two programs that are independent and work together, but still only take CLI parameters etc., so someone else can create a general GUI menu system to interact with the user to conveniently start your backend and frontend programs. This way you'll create a racing game in which parts can be switched (possibly without even recompiling, just by redirecting pipes in Bash), and this extreme moddability and modularity will allow the game to be played on slow PCs, in command line, 8bit mode, as well as on fast PCs with great 3D graphics, light weight as well as heavy weight, and it will give rise to hundreds of different racing games, just as there are hundreds of Linux distros.

The same should apply to engines. Creating an engine as a framework that includes graphics, physics, audio, IO and its own IDE is very wrong. These should be completely independent parts -- libraries -- usable from any language and development environment.

We kind of see this idea working with e.g. roguelikes or games like GNU chess, and editors such as Tiled, but not nearly to its fullest potential, and with not nearly enough games. I'd like to see a big project built like this.

Am I just naive, or is there something right about my thoughts?

[fluffrabbit]

The modular libraries thing exists and works well. Graphics APIs, SDL, GLFW, SFML, STB, physics engines, and small3dlib (which I promise I will work with soon) are great examples of this. I've got my own (unreleased) pet libraries for dealing with rendering, the glTF format, neural networks, audio, action RPGs, etc.

It's an interesting idea to have a modular library for just the gameplay component of a certain genre. Game systems libraries. That sounds awesome, like a higher level version of the C++ algorithm library. Might be tough with a racing game though because many things are 3D-specific. If you just leave out rendering, it's a high-level physics engine with built-in car modeling.

[Julius]

I am a bit torn on this idea. First of all your racing game example would obviously not work as you need a game to test it in, even more general libraries such as a physics engine are hard to get right without a good real world test case. But if you make a racing game, it is probably a good idea to make things modular so that another racing game doesn't have to reinvent the wheel.

But the larger issue is that having a set of libraries can also be an anti-pattern. Look at what happend in the Ogre3D ecosystem. I think they tried what you have in mind, but no one ever came up with a (non-commercial) game engine to tie it all together and thus the maintenance burden (and the burden to make bad choices for libraries) was left with the individual game project, which was IMHO the main reason they more or less all died.

I think it is better to have a monolithic game engine where you can share the engine maintenance burden with other projects. Within that game engine project it makes sense to keep things modular, but to the actual game developers it should be one more or less monolithic package. Already too many plugins etc (see Torque3D) can cause issues, even if the engine itself is quite monolithic.

[freemedia2018]

I'm kind of in the middle here--

Modularity is a good, but libraries for games tend to be pretty specialised-- in my imagination it would be like trying to make a modular oil painting. It's probably just going to be a mess.

I'd be interested in hearing ways to make it work.

[fluffrabbit]

I would like to draw a line between libraries and scripted engines. Ogre is known for being bloated, while Torque is known for just being broken. Ogre projects can work just as well as, say, PyGame projects. You think developers aren't smart enough to make sound decisions about libraries? I would describe a lot of Unity users that way, and those people might use Godot and even Torque, but when you get to something like Ogre I blame the game developer for making stupid decisions because s/he should know better by that point. (PyOgre notwithstanding of course.)

[smcameron]

The finished game is a little bit of an OS in itself -- it has its own GUI system, user profile system, network protocols, built in internet browser, scripting subsystem maybe even with a custom made language etc. This is wrong.

I do not entirely agree with your premise for a few different reasons (I also do not entirely disagree either, but that's not so interesting):

Here are a few thoughts about why free game dev is the perfect place for building weird custom shit:

1. Games are not tools, they are to a large extent art. As art, quirkiness and differences are sometimes sought out. You don't necessarily want your game to look just like everybody else's game, and one way to make it look different is to build, e.g. a GUI system that nobody else uses, or *would* use.

2. Free games especially tend to be hobby projects. A lot of the work done is done for the sake of doing it ... the work itself is part of the fun, maybe even *the main part* of the fun. You're telling me I shouldn't do what I find to be fun in my own hobby project? This is not a teneble position to take. Though I do pick and choose which wheels to re-invent. For example, some people might invent their own scripting language, which I can understand the appeal, it's a pretty interesting project. I chose to use Lua (for a few reasons: a) If I actually want anyone to use it, it should be something which people might already know, b) I'm not confident I could design a language that didn't completely suck, although it would be a *very* interesting problem to chew on, and c) Looking at Lua, and how it would interface with my program, it actually seemed to be pretty good -- this particular wheel did not seem to be too weird for me.) Another example, I did make my own UI widgets. This wasn't necessarily planned, but came from the evolution of my game. In the very early days, I imagined the whole game would be drawn with nothing but lines and I imagined the graphics would be pretty simple. At first, I only needed the ability to draw text via lines, (which I re-used from a previous game), buttons that I could click with a mouse, and some gauges. All easy enough to whip up over a weekend. Over time, I added more widgets in the same style as needed, and reworked the code from some hacky garbage into a somewhat reasonable API. Consequence: my game looks like nothing else out there -- which is a good thing (regardless of whether any particular person thinks it looks good or bad.) I am a fan of forging your own weird wheels for the sake of it in free games.

4. Debuggability. You cannot debug someone else's code as easily as you can debug your own code.

5. Code size. One example: I wrote my own STL, OBJ, (and also oolite .dat) file parser and importer instead of using assimp. Assimp is 300k lines of code (last time I checked) *and* drags in a bunch of C++ dependencies and libraries. My entire game is ~100k lines of code. Should I really use an asset import library that's 3 times the size of my entire game? My asset import code is less than 1500 lines of code[1] -- *two orders of magnitude smaller* than assimp, *and* I don't have to convert from some weird data structure to my game's native data structures. My asset import code uses my game's native data structures directly, and, since I wrote it, I can debug it, and it's not big enough to contain many bugs anyway. It does what I need and not more.

7. Learning. If I design and implement my own lobby server and network protocol instead of using... what raknet? ... then I learn a *ton* of stuff that's useful outside of gamedev, and stuff that's difficult to learn by any way other than by doing something along these lines ... stuff useful in my day job. And this goes for many of the other weird wheels I choose to re-invent on my own.

8. Having reinvented a few of my own weird wheels instead of using someone else's weird wheels, I *have* at least sometimes re-used some of my own weird wheels. By re-inventing my own weird wheels, I am potentially making wheels that other people can use.

9. Because free game dev tends to be a solo activity (not always, and not ideally, but usually) what other coding project is a safer space for a developer to build weird experimental things than games? If the game say, crashes, turns out ugly, or too weird to be understood, what are the consequences of this? Well, we can give the players their money back (Here's your $0 back.) What *other* project would be better for this sort of experimentation than a free, open source game? None. None more better.

[1] I exclude about 2000 lines of the mikktspace code because I'm pretty sure nearly nobody writes their own mikktspace code because it's too hard, and because the code that everybody uses is nicely designed to fit in with anyone's code.

[Lyberta]

Deleted.

[drummyfish]

I hear you all and I respect that people want to make big monolithic projects -- many great tools such as Blender work like this. However, I am frustrated that practically no one tries to do it the Unix way, or even see it is an option, while I really do think it has a lot of potential and in many cases would be the superior approach.

I am not trying to force anyone to do this, I would simply like to find maybe a few followers of my idea here.

I am a bit torn on this idea. First of all your racing game example would obviously not work

Racing game was just an example, but following on the idea, I imagine that besides a classic real time racing game against opponents many other things could be done with it, e.g. a non-realtime racing game that would play like Trackmania and the gameplay would be similar to creating a tool assisted speedruns, and the goal would be simply to search for the best possible time that can be achieved in given track. Or again, the engine could be used in AI research or whatever. It would simply be a general racing component.

So here is another important idea I'd like to communicate here: Make game components as general as possible and don't try to assume any specific use or platform. If you don't need floating point math, don't use floating point. Don't impose dependencies that aren't absolutely necessary for the component. This allows people to use the component in more places and in ways you never though of.

Games are not tools, they are to a large extent art.

Of course games are art, but they are still technology and game development is still engineering. They at the very least can be viewed as tools. E.g. a chess engine is simply a program that helps you handle chess rules, it can be used for research, machine learning, to build chess analysis tools, but also games for pure entertainment. This philosophy could be applied to almost any game.

Free games especially tend to be hobby projects

As many free SW projects in general. That's why we should try to do things efficiently, not create huge monolithic projects "owned" by a few developers (and dependent on them). A big project could simply be a merger of many small hobbyist projects in this case.

Debuggability

The Unix approach is no less debuggable -- I'd maybe argue it is more debuggable, because the components are more separated, you clearly see the communication between the components and any component can be completely isolated and tested with unit tests or in other ways.

Because free game dev tends to be a solo activity

Well but this is exactly what I think is wrong in many cases, this seems to have been adopted from proprietary indie game development, but libre game dev should try to take the full advantage exactly of sharing! Am I not right?

___________________

As @fluffrabbit mentioned, my small3dlib tries to adhere to this philosophy a bit, it is a component that only tries to do one thing: computation of 3D renders, and doesn't do anything else (e.g. actually render to screen), and for this it needs NO dependencies. It also doesn't have external dependencies such as package managers, it's simply a "long snippet" of code that doesn't presuppose any specific use or platform, it only offers to do one type of computation for you.

I would perhaps like to create a game one day that would demonstrate all that I am talking about here.

Imagine an ultimate FPS game composed of many small components our community has created and that allows any player to create a completely unique flavor of it. There can be different distributions of the game for different purposes -- casual players, hardcore players, speedrunners, slow computers, fast computers, children, adults, ... -- just as there are distributions of Linux. You think the opponent AI sucks? Just switch the AI module. The game is too slow? Just switch a module. Whenever a new gaming platform comes out, there will almost immediately be a port of this game to it -- on smart watches, handhelds, VR, ... Just recompile the components, and possibly leave out those that the platform can't use (because of performance etc.). You could do fun challenges like playing a randomly generated flavor of the game. With just a little bit of work you could even try weird combinations like an FPS game combined with a racing game. It can run on a distributed system because the components can communicate using text and they can do that via network. People can use this in education, when learning to code or developing games. Our society needs this!

Here is an example of a projects I used to think about:

A pure text, command-based RPG like Zork, which as such hat NO dependencies, can be ported to small gaming consoles, calculators, toasters etc. However, the text output could be made so as to allow easily adding a graphics front end, and e.g. even a 3D one, with WSAD movement, physics and everything. So instead of writing a command "go through door A", you'd actually move and click the door in the 3D environment, which would generate the command and pass it to the back end. This would of course require modelling 3D scenes and adding extra data and resources that aren't present in the original text game, but it would still work, and it would allow to split a 3D RPG game into two completely independent parts that could be developed and maintained by two independent hobbyists.

[Jastiv]

ah, the dreaded dependency hunt - for a while it was known as "the only game for linux" I hate it when your distro doesn't come with the newest libraries that you need for the game (or worse yet, the game requires some old library your distro has already surpassed.) I have no idea what to do in those situations, and usually I give up. Sometimes I try again with a newer distro, only to end up with the same problem(s).
Java (sort of) solved this with SDKman where you can have multiple versions of stuff for the purpose of development and testing. I also know stuff like snaps, flatpacks and docker are popular, but I haven't experimented much with that, and not every hard to compile game has one of those. (besides doesn't someone have to put it in one of those in the first place?) Really, I'm talking more about how to get abandoned games or those with noob developers to compile.

[Julius]

You think developers aren't smart enough to make sound decisions about libraries? I would describe a lot of Unity users that way, and those people might use Godot and even Torque, but when you get to something like Ogre I blame the game developer for making stupid decisions because s/he should know better by that point.

Well, I think there were quite a few novice developers that started projects with OGRE3D a few years ago, that definitely made bad decisions. But in the end it is also a problem of too much choice and splitting up efforts. e.g. if the audio library that is used in Godot loses its maintainer, there is a high chance that because everyone else is also using it that some new maintainer is found for the benefit of all. But in the case of the Ogre3D ecosystem, if one of the ten or more optional audio libraries has no maintainer then new projects will use another one and old projects are either left with the maintenance burden more or less individually or more likely they will patch around an old version as "good enough" and over time it becomes a huge technical debt.

[drummyfish]

@Jastiv you need to compile libraries from source when your distro doesn't have the new versions.

But generally this is a big problem and it's what I am talking about -- libre games die in the same way proprietary games die because even if freely available, they are existentially dependent and de facto owned by a single person who knows how the code works and how to compile it. When that person loses interest or e.g. the whole game depends on a package that has died and can't easily be replaced, the game dies as a whole, along with all the wheels that were newly invented. It's an immense waste of effort.

[Julius]

Imagine an ultimate FPS game composed of many small components our community has created and that allows any player to create a completely unique flavor of it. There can be different distributions of the game for different purposes -- casual players, hardcore players, speedrunners, slow computers, fast computers, children, adults, ... -- just as there are distributions of Linux. You think the opponent AI sucks? Just switch the AI module. The game is too slow? Just switch a module. Whenever a new gaming platform comes out, there will almost immediately be a port of this game to it -- on smart watches, handhelds, VR, ... Just recompile the components, and possibly leave out those that the platform can't use (because of performance etc.).

I think what you describe is in a way quite similar to Torque3D and it's huge library of legacy addons. It sounds good in theory, but in reality it is a huge mess with outdated and none working plugins & plugin APIs that can not be changed for backwards compatibility reasons. Of cause if plugins would have always been mostly FOSS than the issue would be less acute, but I think it is a general issue. In projects like Blender at least general compatibility of functionality is checked with every release and sometimes they officially drop features that are unmaintained.

[Julius]

But generally this is a big problem and it's what I am talking about -- libre games die in the same way proprietary games die because even if freely available, they are existentially dependent and de facto owned by a single person who knows how the code works and how to compile it. When that person loses interest or e.g. the whole game depends on a package that has died and can't easily be replaced, the game dies as a whole, along with all the wheels that were newly invented. It's an immense waste of effort.

I think you are looking too specifically at games (proprietary games usually "die" the day they are released and get only minimal updates afterwards, a few multiplayer titles and notable exceptions aside). What seems to be the core of your concern is I think general maintainability of large code-bases over generations of developers... something we are starting to see only as an issue since a few years and which is why large projects like Windows or SAP are showing visible signs to crumbling under their own weight/legacy code-base & complexity. In our smaller world of games this is more visible with long running FOSS game projects as they usually die more noticeably due to such issues.

I recently read an interesting article on how the GO language architects at Google specifically designed the language around such maintenance concerns and development with multiple generations of often not very experienced programmers. The language seems to get a lot of flak for the choices they made deliberately to avoid some of the issues raised here though. I'll try to find the link to that article again. Edit: maybe it was this one: https://yourbasic.org/golang/advantages ... va-python/

[freemedia2018]

I am not trying to force anyone to do this, I would simply like to find maybe a few followers of my idea here.

I think the idea has potential, but it might not have as much potential as you realise.

So here is another important idea I'd like to communicate here: Make game components as general as possible and don't try to assume any specific use or platform.

That's going to be a hard problem.

Games are not tools, they are to a large extent art.

Exactly.

Of course games are art, but they are still technology and game development is still engineering.

Yes, but what I think you're missing is that they're engineering "second."

It's hard to make new tools that people will choose over old tools, for a highly creative process, unless the technical side is the focus.

Maybe with games, the technical side is not the focus. I can think of a few reasons why that could be.

This philosophy could be applied to almost any game.

But perhaps not as easily as with other software projects.

A big project could simply be a merger of many small hobbyist projects in this case.

I like that idea. It's difficult though, because when you get highly creative types they have "creative differences." It's hard for a single project to cater to that.

Because free game dev tends to be a solo activity

Well but this is exactly what I think is wrong

I think it's natural. Actually most development is a solo activity.

It's a plus to get people to work together towards a common cause, but the more "distinct" (e.g. with art) that a project tries to be-- I'm the second person to mention it-- the harder it is to create toolkits for someone looking to make distinct things.

What I'm definitely not saying is that you should give up on this idea or that it can't possibly work. I think it's a lot more difficult to prove, and without outlining more about these components, you won't have a chance I think.

I'm always trying to get people to build freedom labs. This is a perfect reason for one-- you need to do research along these lines if you want to convince people and promote this.

A new libre gaming movement and strategy and philosophy. But above all, for the reasons you're being given here (I'd urge you to look at them as valid, not as excuses-- this feedback will help you figure out just how much it would take for your idea to work) it's not going to be easy.

If you're too shy to create a free software organisation for anticapitalists, perhaps a freedom lab for libre gaming components (and this overall idea/concept/philosophy in particular) would be something youre better suited to. Only you would know for certain. I'd encourage you to explore it, just know it's going to be a long march uphill. Not everything worth pursuing is straightforward or easy. This won't be. Best of luck, let me know if I can help somehow. Though the feedback you're getting here is quite valid IMO.

[fluffrabbit]

if one of the ten or more optional audio libraries has no maintainer then new projects will use another one and old projects are either left with the maintenance burden more or less individually or more likely they will patch around an old version as "good enough" and over time it becomes a huge technical debt.

That doesn't sound so bad if the game developers can maintain the libraries once they become abandonware.

[drummyfish]

Games are not tools, they are to a large extent art.

Exactly.

This is a very fun argument, it fascinates me because I have heard exactly this argument in other contexts from other people, and it never makes sense to me. E.g. one person told me that games are an exception to free software principles, because they are a form of art. I fail to see any sense in this form of art argument.

Firstly, everything is eventually art -- tools are art, good UI is art, creating good UX for tools is art, a good CLI is an art and good engineering and design itself is always art. The only thing that isn't art are trivial things that don't require creativity, like writing a bubble sort for the thousandth time, but games to me are no more a form of art than any other similarly complex computer program. I personally can't see how they could be separated this way.

Secondly, good design doesn't stand in they way of artist any more than bad design does.

Thirdly, even if hypothetically you had to make a choice between good design and good art -- e.g. because of limited resources -- I'd really love to see some people say "let's put design first." I understand the consumers will always prefer art, because design is invisible to them, and for this reason for profit games will also put art first, but in a community of hobbyist developers I wouldn't expect to see exactly the same. There is a great lack of balance here in my opinion. I would expect at least a few people in this community to focus on good internals.

[Julius]

if one of the ten or more optional audio libraries has no maintainer then new projects will use another one and old projects are either left with the maintenance burden more or less individually or more likely they will patch around an old version as "good enough" and over time it becomes a huge technical debt.

That doesn't sound so bad if the game developers can maintain the libraries once they become abandonware.

It all adds up, and do you want a game developer to focus on developing a game or maintain a audio library? I am willing to agree to the latter if it is part of a larger engine project that benefits many game projects, but if it is only for a single game project it is hardly better than writing your own audio library from scratch.

[drummyfish]

But actually now I think I kind of see your point -- there is this cultural idea (and again I blame the proprietary, for-profit games for it) that the goal of a game is always to appeal to as many users as possible, and this means optimize the appeal to nontechnical users, because these are the majority, and these are also unable to see the art of design, so we optimize audiovisual art. This idea I think is manifested in the form of art argument.

My complain in this context is that no one sets a different goal with their game, or even realize they could. The different goal here could be e.g. to appeal to both technical and nontechnical users alike, to make the game survive and spread to as many platforms as possible, to spark additional creativity and so on, even if this means fewer users in total.

[freemedia2018]

Modularisation helps keep free software free-- particularly when we don't want to keep reinventing the wheel.

To some degree, this also helps people making games. But in games, the fact that it is more "art" than usual means the wheel is going to be reinvented more often. Which means modularity helps less.

If you remain fascinated, keep pushing for what you think is better. Just know that it's going to take a lot of pushing, and a lot of examples-- not just a good argument. Since you can code, this is something you can do. But by all means, keep looking for believers. Happy to help with that, I just think it will be slow to start (and probably won't ever convince all gamers to use more modular tools.) That doesn't mean you're wasting your time.

[fluffrabbit]

So many textwalls.

I would argue that maintaining internals that somebody else made is less work than writing your own from scratch, assuming you found the internals in good condition and fixed things as time went on. It's not an ideal situation, but it works for games and most "modern" software if you get my drift.

[freemedia2018]

So many textwalls.

Purely as a practical suggestion, if you tire of longform conversation, just add the offender's username at the end of this link:

https://forum.freegamedev.net/ucp.php?i=zebra&mode=foes&add=

And click on it.

No more textwalls, everybody's happy. Some people are detail oriented, it's not likely to change.

[dulsi]

Unix had the advantage of taking output as one tool as input to another. The first command is generally just running. It isn't usually accepting more input by the user. Games aren't really as nice and simple. You have to have some sort of interface between the components that is more complex. Frankly I think this is already being done with engines and libraries. It's not as separate as you wish.

[fluffrabbit]

Some people are detail oriented

Not you, apparently. Thanks for the tip anyhow. It's def. going to improve my productivity.

[smcameron]

The Unix approach is no less debuggable -- I'd maybe argue it is more debuggable, because the components are more separated, you clearly see the communication between the components and any component can be completely isolated and tested with unit tests or in other ways.

I have to disagree with this. The particular example I gave was assimp (~300k lines of C++) vs. 1500 lines of C that I wrote myself. The 1500 lines of code I wrote myself is going to be at least a million times easier *for me* to debug. And you seem to somehow assume that code I wrote cannot also be debugged in isolation and is not clearly separated. And debugging my own code is almost fun. Debugging someone else's C++ is misery. And if I did use assimp, I'd probably have to write around 1500 lines of code to convert data from assimp's idea of how it should be to my game's idea of how it should be anyway, and I'd have to debug that too.

[fluffrabbit]

Eh, that's not a great argument. There are smaller and more specific libraries for specific formats. Assimp is almost a game engine, and it has more lines of code than all of my game engines put together. The STB-style libraries really shine in this area.