hasufell {l Wrote}:hiker {l Wrote}:[*]they often break the API from one version to the next (bullet), and atm for example we do not even use (nor support) the latest bullet version.
That's not an argument at all. Distros have means to handle API breakage of libraries. (soname, any1?)
That should not be your concern at all and you can blame distributors to fix this.
I am not blaming distributors for that (or anything). But I agree, different versions can obviously distinguished if the distributor are willing to maintain that many versions of a library - I was more thinking of the followup problem: I can't see a good reason for us to spend effort to update bullet at this stage - it would just make a lot of work (porting, debugging), for no gain (unless we encounter a bug, or need a new feature). So for all I know STK might still be using bullet 2.79, when bullet reaches the 5.0 milestone
Would you still be willing to maintain bullet 2.79 at that time - esp. since each project might need its own special version of bullet? You are talking of wasted time further below - in these circumstances (admittedly somewhat theoretical, I don't think you will have that many applications that use bullet), would you prefer having to re-test X versions of bullet every time a new compiler is used for a release? As opposed to us just maintaining and managing this for you?
Re bullet some examples: we had to apply quite a few changes, which would certainly not be accepted upstream:
- we had to remove SSE support (since it causes crashes - likely because some memory-misalignment in STK, which we never fixed, but since bullet performance is not an issue atm it's not important for us)
- apply some other changes which are more or less STK specific (e.g. made some variables protected instead of private - though admittedly that might be accepted upstream)
- I also remember that one physics process was changed in a new version, which might be 'more correct', but just broke STK. So we had to switch this new implementation off, another compile time change. This would be completely unnoticed (till you try to drive our karts, and perhaps then even under some rare circumstances).
- one file in bullet broke when compiled with a certain compiler revision with default optimisation flags. We applied a work around (switched off one optimisation feature) for that and reported it - but still, your system libraries might just trigger that bug again. Bullet will potentially fix it in a new release, but in my experience won't release a bug fix version.
Generally, I don't think it's a good idea to link any program against a library it wasn't developed and tested with.
hiker {l Wrote}:[*]we apply bug fixes. Those are reported back to the developers, but might not be fixed before the next release of the corresponding library, which is too late for us (e.g. we only recently fixed a memory leak in bullet). Using a standard bullet will result in memory leaked constantly in each race.
That's understandable, somehow. However you should document that in the tarball (in README or INSTALL file) with a reference to the upstream bug report, so that packagers can be aware of the situation and unbundle it if your patches made it upstream.
Generally those things are documented in our commit log. Just get the log for the bullet/irrlicht tree, and you will see our changes with the reason for it. The way you suggest is actually more work for you: you have to keep track of any patches we apply, while otherwise we will do this for you
This should always be considered an unresolved bug.
hiker {l Wrote}:[*]we need changes to compile-time constants (irrlicht), which the developers are not willing to make in their source code (and they recommended to use our own version)[/list]
That's the only real problem so far. There are solutions, but I can understand that these might not be viable:
1. fork it
2. switch to a different library (is there even one?)
You know _any_ bug free library? OK, there might be one or two (libhelloworld anyone?) - but suggesting to switch to a different library (even ignoring that it took us one year to switch in the first place) is just nonsense
I know that these suggestions might be completely unrealistic.
Forking isn't, which is more or less what we did. Though I admit I don't like forks (it just means that development effort is duplicated, instead of all working towards a common solution ... ok, there are of course valid reasons for forks ... I don't want to get sidetracked here
). We hope that we can stay as close as possible to official irrlicht - and perhaps one day they might even remove the hard-coded limits we need to change
Let me just sum up why bundling is bad and should be avoided if possible. I don't expect you to do something about it right now, but I hope that you still consider this situation a "bug":
- security: bundled libraries are somehow out of the scope of packagers. zlib had many severe vulnerabilities and if packages bundle copies of zlib, then every single one of it needs to be fixed _manually_. If packages link to it, we just fix up the system lib and are done. Also note that you (via Irrlicht) currently also ship a bundled copy of zlib.
These things are difficult to track and will cause security issues on the long run, cause developers/packagers and users might not even be aware that they are affected.
First of all, we re-package all zip files that will ever be used by irrlicht's included library. So even if a bad zip file would get submitted, it should be detected while re-packaging (with up-to-date zlib libraries). But I agree that this might be a concern for you, so just use:
- {l Code}: {l Select All Code}
-DNO_IRR_COMPILE_WITH_ZLIB_
for your compile flags, and you are done. Similar flags exist for other included libraries. Note that one of the big advantages of irrlicht (at least from my point of view) is that it included many third party libraries, which makes it a lot easier for us to get new users to compile STK (less dependencies).
maintenance: for you and for us. You modify it, we have to track down security issues, QA issues, build time issues and plain bugs not just for the system lib, but also for every modified copy in every possible package. So that you not think I'm talking stuff... this is already true for Irrlicht, cause the Makefile is messed up in several places and ignores users CFLAGS. I will send parts of those patches upstream anyway, but it then has to reach your bundled copy as well, maybe it does not or is even incompatible, cause your version is modified.
We would welcome any patches that fixes that (the patch for FindFribidi.cmake is already applied - thanks for that!). Irrlicht development can be rather slow (one reason why we had to use an unreleased version before, and finally decided to maintain our own version). But yes, it is additional overhead, but I don't see any other choice.
support: experience shows that people who bundle stuff don't accept bug reports for their bundled libraries although they might have modified them heavily (popular example is freeorion which "forked" gigi, but does not support it at all). Take it up with upstream they say. When I do it, upstream will tell me: erm, we don't support random forks. Maybe your version even introduces new bugs. No one will be responsible. This decreases support.
I will gladly apply any patch that affects STK (assuming that it is also submitted upstream - we want to stay as close as possible to irrlicht).
size: increased size, both in memory and on the disk, obvious
I can't see many people having two irrlicht applications running at the same time tbh - besides, disk space nowadays is not really an issue (for this kind of memory - stripped libIrrlicht.a is 14MB).
hiker {l Wrote}:So please, do not go this way, you will create a buggy version of STK, which will make players unhappy with STK, and causes us problems by people reporting the bugs you have (re-)introduced. If you do not want to support our included library, then please do not include STK, since your version will be buggy.
Oh, I forgot... this is the 5th reason not to bundle libs
People will break your application and apply lolpatches.
We are using static libs, so I admit I am not sure what you mean here.
hiker {l Wrote}:Instead ask your users to download our linux binaries.
No.
The bundled libs are no reason for Gentoo to not include STK, but if the situation gets worse, then packagers might just not care enough to spend time on it and keep track of the situation.
I fully understand this. If irrlicht should be improved to get rid of the compile time constant (not sure if this is possible, I just assume it could be done), we might consider it (though having had problems in the past with the speed of irrlicht development I am not guaranteeing anything here
). Then again, I've already seen some code I wanted to re-use (caching of meshes), but I can't without some changes to irrlicht, which I am not sure the developer would accept (but I will certainly try).
Besides this rant: STK works nicely and I see many gameplay improvements. Keep it up.
Thanks a lot - not only for adding STK to Gentoo, but also for the time to tell your side of the story. In an ideal world I would gladly agree with most what you wrote - but unfortunately at this stage I don't see any other option that will guarantee a bug-free experience for players.
Is there anything we can do to help you? Should we include any config files or so - e.g. to make it easier for you to disable the included png/zlip/jpg libs of irrlicht?
One other quick question: do you have any stats about the number of downloads of STK from your servers? I would be interested to know how many linux downloads there are that we don't know about.
Cheers,
Joerg