Github & data along source code

[Bertram]

Hi everyone,

Since the issue tracker is officially dead, I'm taking the opportunity to open a new thread here so we can discuss the following topics, which have been in my twisted mind since some time already:

- The potential use of github as a development reference.
Why am I proposing this?
Because github has got an integrated issue tracker, permit code reviews and in-lines comments, and it is way clearer and more responsive when it comes to review code.

Github popularity will also permit to attract new developers more easily.

As a proof of everything stated above, hwoarangmy, MCMic and myself are already using my github fork as a reference and I only have the surplus work to resync with sourceforge from time to time.

Last but not least, new (and old) developers willing to help the projects will have to fork it and make use of the pull requests workflow very famous on github. hwoarangmy and me are very used to that already, permitting to make sure a review happened before some code is pushed to the main repository. I'll be honest, I'm looking at Paul at this very moment.

- The migration of data files along the source code to ease commits and overall development.
Please note that I'm speaking of game data directly used by the engine, not their source files. I.e.: blend files can stay in the SVN media repo, while the mesh files are used by the engine and thus are part of the game data.

Why? Well, it's another thing that is surely making hwoarangmy use the github fork instead. Indeed, he doesn't have to split his commits between the data files and the source code, and it's making the commits much more logical, since the upgraded game data goes along the upgraded code.

Also, when one developer forks the project, he/she doesn't have to copy the media files or create a symlink to the very heavy media repo, not getting diffs on data files, and such mess, making contributions there very difficult. If we want to attract new devs/artists, we cannot afford making the collaboration difficult.

I really think this move would help the project, especially as certain members of it are already working that way for the reasons given above.
Now, I'd like to get your input.

Best regards,

[Akien]

For the data file, you can also use a separate github repo (e.g. github.com/opendungeons/opendungeons-data) and use git submodules to reference the location of the mesh files into the main tree (github.com/opendungeons/opendungeons). One then has to clone the main repo and update the submodules to grab the game data. This way you can also upload the .blend files to github and link only the meshes as submodules (the main advantage would be to have everything at the same place).

It also makes it possible to provide the source code and the data separately for packaging purposes. It's especially useful if the game data is huge, since it can then be packaged (at least with RPM) as a architecture-independent package, so it is not duplicated in 32bit and 64bit repositories contrary to the game binary.

Note that there are tools to convert SVN repos into git repos and to keep the commit history, that would be useful if you switch to github.

[MCMic]

I support using git.
I’m not sure about using github, it being centralized and all. But I don’t really care, and as I won’t help putting anything else together I don’t really have that much to say about the hosting part.

[Bertram]

Hi guys,

@Akien:

For the data file, you can also use a separate github repo (e.g. github.com/opendungeons/opendungeons-data) and use git submodules to reference the location of the mesh files into the main tree (github.com/opendungeons/opendungeons). One then has to clone the main repo and update the submodules to grab the game data. This way you can also upload the .blend files to github and link only the meshes as submodules (the main advantage would be to have everything at the same place).

Well, I'll be honest, if I were to use submodules for the data, I wouldn't ask people to put those along the engine code in the first place since the point is to permit easier commits mixing data + code. (Yes, it's much easier working that way, IMHO. ;])
I can understand your point about losing history and thus author information about the data files in that case, that's why we should indeed create some author file where credits is given where credits is due, anyway, and that we'd keep the svn repository for the source files at least.

It also makes it possible to provide the source code and the data separately for packaging purposes. It's especially useful if the game data is huge, since it can then be packaged (at least with RPM) as a architecture-independent package, so it is not duplicated in 32bit and 64bit repositories contrary to the game binary.

Hmm, I disagree there, since first it's the packager problem not the developer's one. (Owww, Bertram is sooo evilish. ;P) (joke). More seriously, the point is more about configuring cmake to create clean and different packages if packagers need it, more than splitting the packages by other means, in order to, once again, ease the developer's work. Also, OD is not some kind of engine aimed at only being used for total conversion but rather a full game. This means data is strongly linked to the current engine version and separating the engine from the data is only relevant for fixing the binary packages, which 99% of the time, triggers a fixed game data package anyway, or small engine bugfixes.

@MCMic

I support using git.
I’m not sure about using github, it being centralized and all. But I don’t really care, and as I won’t help putting anything else together I don’t really have that much to say about the hosting part.

Well, we're already using git (at sourceforge) for code and the point is not really to use git for data but rather have game data along the source code to permit mixed commits with both data and code as it is much easier, experience made so far. Sad that you're not planning to help anymore, though.

All that I said above is only my opinion. If you have a better experience than me with submodules or something, for instance, feel free to prove me wrong. I'd me more happy to find some way I didn't know of or didn't think about to improve coding.

Best regards,

[hwoarangmy]

As a proof of everything stated above, hwoarangmy, MCMic and myself are already using my github fork as a reference and I only have the surplus work to resync with sourceforge from time to time.

I use it because you do But I agree it is pretty easy to use and have usefull tools.

Please note that I'm speaking of game data directly used by the engine, not their source files. I.e.: blend files can stay in the SVN media repo, while the mesh files are used by the engine and thus are part of the game data.

I also think it would be a good thing to have only 1 repository to download to compile the sources. The more complex it is to download sources (I mean downloading dependencies, sources from different locations, ...), the less chances we have to get new coders/artists.

I have not enough experience to say if github is better than the others. But I played another open source game lately called UFO AI (a XCOM clone). They use git on source forge. They have a very large repository (more than 1Gb). And from what I read on their forum, they are not happy with SF. I have also tried to download their sources and it crashed several times. That being said, I don't know if there are similar problems on github with big repositories...

[MCMic]

Sad that you're not planning to help anymore, though.

I was only talking about web hosting and organisation, I do plan to keep helping with coding when I can.

[Akien]

It also makes it possible to provide the source code and the data separately for packaging purposes. It's especially useful if the game data is huge, since it can then be packaged (at least with RPM) as a architecture-independent package, so it is not duplicated in 32bit and 64bit repositories contrary to the game binary.

Hmm, I disagree there, since first it's the packager problem not the developer's one. (Owww, Bertram is sooo evilish. ;P) (joke). More seriously, the point is more about configuring cmake to create clean and different packages if packagers need it, more than splitting the packages by other means, in order to, once again, ease the developer's work. Also, OD is not some kind of engine aimed at only being used for total conversion but rather a full game. This means data is strongly linked to the current engine version and separating the engine from the data is only relevant for fixing the binary packages, which 99% of the time, triggers a fixed game data package anyway, or small engine bugfixes.

That's true. The only nuance was that arch-independent packages are uploaded only once to mirrors, while arch-dependent packages are uploaded for each arch. So if the data (e.g. 300 MB) is bundled with the binaries, it will take 900 MB of space on each mirror (32bit + 64bit + source package), while it takes only 600 MB if it's a noarch package (noarch + source package).

But my argument was wrong, because I can create arch-dependent and arch-independent packages in the same source package, i.e. using the same source tarball. And then your argument that most of the time the source code changes, the data changes too, invalidates my whole point :P
Evil Bertram won the fight, but not the war! Archlich Akien will strike back!

[Bertram]

Hi

I was only talking about web hosting and organisation, I do plan to keep helping with coding when I can.

Ah cool. \o/

But my argument was wrong, because I can create arch-dependent and arch-independent packages in the same source package, i.e. using the same source tarball. And then your argument that most of the time the source code changes, the data changes too, invalidates my whole point :P
Evil Bertram won the fight, but not the war! Archlich Akien will strike back!

I fear this will have to be settled out on the battlefield, you Archlich-ish creature. I can't wait to send my crawling minions to your dungeon impoverished defenses. Mwahahaha.
This, alone, adds some good reason to improve OD and fix a few bugs. ;D

I use it because you do But I agree it is pretty easy to use and have usefull tools.

Oh, thanks for the courtesy.

I have not enough experience to say if github is better than the others. But I played another open source game lately called UFO AI (a XCOM clone). They use git on source forge. They have a very large repository (more than 1Gb). And from what I read on their forum, they are not happy with SF. I have also tried to download their sources and it crashed several times. That being said, I don't know if there are similar problems on github with big repositories...

Just to point it out, they have started migrating to gihub: https://github.com/ufoai/ufoai

It's true that cloning big repo (even shallow ones when using --depth=1 and then using fetch for the next depths) requires some bandwidth or will timeout on git. The ability to resume a clone/pull is very missing, IMHO, but not simple to add.
That said, I think it's not our case at least at the moment. And, of course, I'm not speaking of data source files which should stay on SVN.
There are much bigger repositories than ours so I think we have time before facing such problems.

Regards,

[Bertram]

The main useful thing with sf.net has been the web hosting, though I suppose it wouldn't be that difficult to replace that either if there was a strong desire to move somewhere else.

I don't mind changign to github or other source hosting site, if it is true that it would attract more people . I don't mind even if things get more complex.
Also : so your point is to move away entirely from the SOURCEFORGE , right ? What I can say, for me before joining OD it was a way of quickly monitoring what happens in the world of open source games .... but for now we cannot exceed 100 downloads / week , so maybe it's not a big lost ...

Good to hear, and I do think it would make things easier.
But no, my point is not to move away from sourceforge completely. The media source (i.e.: .blend files and such) should stay on the media source SVN repo, IMHO.
Sourceforge should still be used to distribute compiled binaries, and to host the website. I'd only use github for what it is good for, ease the coding effort, and why not mirror the tarballs.

[Akien]

How about moving the blend files to github too (in a subrepo, see for example what our neighbour does at https://github.com/stuntrally ), and use Sourceforge only to distribute source tarballs and compiled binaries?

I've seen many projects do this, it has the advantage that contributor don't have to learn how to use git + svn but can concentrate on git. I'm not a strong advocate of this though, I just wanted to add this possibility to the discussion.
The main advantage is that everything is in one place (github), though in separate repos (one repo for source code + meshes, one repo for assets source). I also experienced github to be more reliable than sourceforge to generate and download tarballs of the repos. On the other hand keeping the blends on SVN is not a bad choice either, because it's already there, and for such a repo you don't need a deep understanding of how SVN works (it's basically just checkout, add and commit that would be used).

[Akien]

I have not enough experience to say if github is better than the others. But I played another open source game lately called UFO AI (a XCOM clone). They use git on source forge. They have a very large repository (more than 1Gb). And from what I read on their forum, they are not happy with SF. I have also tried to download their sources and it crashed several times. That being said, I don't know if there are similar problems on github with big repositories...

I confirm that it's practically impossible to download big snapshots on Sourceforge, most of the time you have to do a good ol' svn checkout and generate your source tarball (direct download of upstream tarballs works fine though, it's only the "repo snapshot" function that is unreliable in my experience).
With github it works alright most of the time, the only thing that annoys me is that it proposes only to download zip files and not tarballs (but you can copy-paste the path to the zip file and change "zip" to "tar.gz" to get a tarball).

[Bertram]

Hi Akien,

How about moving the blend files to github too (in a subrepo, see for example what our neighbour does at https://github.com/stuntrally ), and use Sourceforge only to distribute source tarballs and compiled binaries?
I've seen many projects do this, it has the advantage that contributor don't have to learn how to use git + svn but can concentrate on git. I'm not a strong advocate of this though, I just wanted to add this possibility to the discussion.
The main advantage is that everything is in one place (github), though in separate repos (one repo for source code + meshes, one repo for assets source). I also experienced github to be more reliable than sourceforge to generate and download tarballs of the repos. On the other hand keeping the blends on SVN is not a bad choice either, because it's already there, and for such a repo you don't need a deep understanding of how SVN works (it's basically just checkout, add and commit that would be used).
[...]
I confirm that it's practically impossible to download big snapshots on Sourceforge, most of the time you have to do a good ol' svn checkout and generate your source tarball (direct download of upstream tarballs works fine though, it's only the "repo snapshot" function that is unreliable in my experience).
With github it works alright most of the time, the only thing that annoys me is that it proposes only to download zip files and not tarballs (but you can copy-paste the path to the zip file and change "zip" to "tar.gz" to get a tarball).

All of this is true. To be honest, I didn't want to ask people to move everything at once at first to avoid frustrating everyone by a whole move request that would break the habits of even the artists, which are not devs, and usually aren't as fond or used of versioning systems as code guys. But also because I joined this project still recently and wasn't even planning to code for it at first, so I wouldn't like people to see me as new wanna-be evilish dictator that wants to print his new rule, even if in the end, I act as one when I request such things.
My point is to get better tools to control the development flow, and review code, ending in better quality for the project. It's just I'd like OD to happen as a playable thing. Then, you will be able to crush me online, on dark and damp galleries dug by creepy monsters... or not. ;P

As for the blend files and such, what you said is true, and could even be done in a second step if not at first, but I'd like to get input from artists, such as Elvano, Danimal, Skorpio, ...

Regards,

[Bertram]

As it seems dev people are ok with the move of code to github, I've created the new github corresponding organization: https://github.com/OpenDungeons

If someone has got something against, it is now time to tell it.
I've created access groups and added people I knew were on github. If you've got an account, please tell me what it is, and I'll do the rest. Please note that I won't grant push access by default to anybody, as the goal of this is to make people do merge requests, and make everyone/any other developer to review them before merging the code by a dev at ease with the project and git practices.

I would also like to grant access to at least another trustful admin so the organization can live on when I'm not around. My votes would go to Danimal and Akien or MCMic.

Regards,

[Danimal]

Im fine with the moving, to me its not much of a change since i dont have to push code. But im clueless with it, so im grateful for your thrust vote but some coder with experience on it would do a better job, anyways, if you think its a good idea i can hold admin access in case everyone goes MIA, so i can act as a fallback option.

You people are forcing me to learn horrible things , where is some good tutorial on Githhub? i think its time i learn to push my crap around, i cant depend on you all forever...

[Bertram]

Hi Danimal,

Don't worry, I won't force you to push pull and whatever right away, it's just as you said, someone to hold the fortress in case there are people away. Please, do create a github account and I'll grant you access. Github has got a lot of doc there as for git but the rest is almost always self-explanatory: https://help.github.com/

Best regards,

[Danimal]

Im fine with it, once i learn to handle Github ill give a good cleaning to the model repository, you would be amazed of what kind of useless crap is still stored in there.

[Bertram]

Eh eh.
Ok, shall I create a OpenDungeons-media-source repo there then? (for blend file and such?)

[Danimal]

Sure, if thats what everyone agreed. My Github user is Danimal696, i have also wondered a few times if i should upload my blends as well and not only the .mesh

[Bertram]

Ok, I've created the media-source repo and the artists team.
https://github.com/OpenDungeons/OpenDun ... dia-source

You are now an owner of the project OpenDungeons which gives you the possibility to destroy anything, so please take care.
I've also made you part of the Artists team.

The repository hasn't been initialized yet but I plan add the mediaSource part of it from the SVN repository. IMHO, if you add the mesh file to the openDungeons repo, you won't need to add it to the media source one. if you have troubles making merge requests for OpenDungeons, you can always do as you always did: Post it on the forums and a dev will take care of it.
As the model source repo, I think it would be good if you can manage it, but let's no go too fast.

Feel free to update the SVN repo with whatever cleanups are needed and I'll import the media-source part afterwards. I'll also sync the OpenDungeons repo with the cleanup made on the media part. (The part about game data used directly by the engine.)

Regards,

[Akien]

Feel free to update the SVN repo with whatever cleanups are needed and I'll import the media-source part afterwards. I'll also sync the OpenDungeons repo with the cleanup made on the media part. (The part about game data used directly by the engine.)

Do you plan to convert the SVN repo to a git repo (i.e. preserve the commit history) or simply to upload the media sources to the new repo as first commit?
I know a Mageia sysadmin who converted all our SVN infrastructure to git, so I could ask him for some help if you want to try to preserve the history of the SVN repo (but maybe you already know how to do this ).

[Bertram]

Hi again Akien,

Do you mean something like this? https://help.github.com/articles/import ... subversion

Well, I can do it but that depends, since the objects there may be really big, I wondered whether importing the history would make the git repo already unclonable for low bandwidths. what would be your advice?

[oln]

Is there any reason we would really need to clone the history from svn repo?

[Danimal]

You are now an owner of the project OpenDungeons which gives you the possibility to destroy anything, so please take care.

*Glup* Ok...

Lets go slowly then... ill clean the SVN repo first, do i need permission as well to do it or is it the same user as Github?;
once thats done you will sinc SVN with Github and i can add(pull request) models directly to "openDungeons repo" or "media source", being media source kinda my playground. Is that all correct?

[Bertram]

*Glup* Ok...

Don't worry, but don't click on red buttons on github.

Lets go slowly then... ill clean the SVN repo first, do i need permission as well to do it or is it the same user as Github?;
once thats done you will sinc SVN with Github and i can add(pull request) models directly to "openDungeons repo" or "media source", being media source kinda my playground. Is that all correct?

Err, nope. this is the SVN repo you'll need to clean. You'll need the sourceforge credentials to make changes there.
http://sourceforge.net/p/opendungeons/s ... ree/trunk/

The media/ part should be the game data
and the mediaSource/ part should be all the working files, blends, svg, ...

Once you've cleaned up things there. I'll initialize the github media source repo with SVN mediaSource/ and sync the github opendungeons repo with what has been changed in SVN media/.
The remaining question is whether we should keep the history for the mediaSource/ part.

I think not but the decision should rather be yours, IMO.

EDIT: Once the repo will be initialized, you'll be able to clone it. One of us devs will then help you if needed. Note that the artists group can directly push the media source repo, as I don't think it is extremely important to remove that right from you guys while it is for code devs that can break things much more easily.

Regards,

[hwoarangmy]

I've found a game called Zero-K. I never tried it yet but it seems to be one of the best open source RTS available and they use github.
http://zero-k.info/

When I have to choose between platforms I don't know, I usually look at what others do. If UFO AI and Zero-K use it, it is probably not a bad choice for us
And it would be interesting to see how they handle synchronisation...