Creature/object definition files

[oln]

After messing with shaders for a while, and looking at some other projects, I came to the conclusion that we should try to start making files to define different creature types, rather than specifying them in the map files. (This has been the plan as far as I remember from talking with andrew long ago.)

So we have to decide what these files should contain, and how they should be structured.
Structure:
Should we use XML, another markup language, or just a plain text file that we parse ourselves?

What do we need the file to contain?

• Mesh name
• material names/texture names/render effect
• Sound effects - filenames, what action they are bound to, and delay/synchronisation info to synchronise the sound to animations
• Creature type, name and statistics
• AI script filename? (if we start allowing scripted creature AI's at some point)

[StefanP.MUC]

For plain definition files I strongly suggest something that is already a common standard, like XML.

[svenskmand]

Yes xml is the solution for that kind of job. Also it would be nice to have a editor to be able to syncronie sounds to mesh animations.

[oln]

First draft/suggestion for the structure (Can a mesh have more than one material? Otherwise the material section could be changed a bit):

{l Code}: {l Select All Code}

<?xml version="1.0" encoding="UTF-8" ?>
<creaturetype name="foobar" meshname="foobar.mesh" job="BasicWorker"
bedname="foobarbed">
<!-- etc -->
<materials>
    <!-- For most creatures/objects we just want a normal mapped or per pixel lighting material, and specify textures, and optionally lighting parameters -->
    <generated name="foobarmaterial" type="normalmap_multipass" space="tangent">
        <texture type="diffuse">
            foobar.png
        </texture>
        <texture type="normalmap">
            foobar_nm.png
        </texture>
        <specular>
        0.5 0.5 0.5 1 20
        </specular>
        <diffuse>
        1.0 0.5 0.5 1
        </diffuse>
<!-- etc -->
    </generated>
    <!--
    <custom name="foobarmaterial" filename="foobar.material />
    -->
</materials>
<sounds>
    <sound filename="foobar.ogg" animation="dig" delay="0.2" />
</sounds>
<!-- <ai filename="foobarai.as" /> If we make creature ais scriptable at some point-->
</creaturetype>

Current structure in map files:

{l Code}: {l Select All Code}

# className   creatureJob   meshName   bedMeshName   bedDim1   bedDim2   scaleX   scaleY   scaleZ   hp/level   mana/level   sightRadius    digRate   danceRate   moveSpeed   cHumans   cCorpars   cUndead   cConstructs   cDenizens   cAltruism   cOrder   cPeace

Wyvern   WeakFighter   Wyvern.mesh   Bed   2   2   1   1   1   5   5   15   0   0   1.3   1   0   0   0   0   0   1   0

Should we use attributes to define all the parameters here, or does anyone have a better suggestion?

[StefanP.MUC]

Looks good.

Attributes are good, IMHO:

{l Code}: {l Select All Code}

<attributes danceRate="2" bedDim1="3" />
#and so on

In the end we should also provide a graphical editor for the creatures (maybe directly from the game), then the user won't have to care about the internals anyways.

[Knitter]

You can use attributes for whatever you want but they were created for one single purpose: meta-data.

Attributes are meant to give information about the data stored not to store data themselves, everything that is relevant should be an element. Still, nothing prevents you from putting data in attributes instead of elements.

Personally I feel that using attributes for data is wrong, but may be due to having worked with XML for quite some time and used some libraries that didn't support attributes.

[svenskmand]

Looks pretty good, but I think we should keep values as attributes and only use elements/nodes to give structure to the file, here is you filed corrected according to this:

{l Code}: {l Select All Code}

<?xml version="1.0" encoding="UTF-8" ?>
<creaturetype name="foobar" meshname="foobar.mesh" class="BasicWorker"
bedname="foobarbed">
<!-- etc -->
    <materials>
        <!-- For most creatures/objects we just want a normal mapped or per pixel lighting material, and specify textures, and optionally lighting parameters -->
        <generated name="foobarmaterial" type="normalmap_multipass" space="tangent">
            <texture type="diffuse"/ filename="foobar.png"> <!-- There can be multiple of these -->
            <texture type="normalmap" filename="foobar_nm.png"/>
            <specular value1="0.5" value2="0.5" value3="0.5" value4="1" value5="20"/> <!-- I did not know what to call these attributes -->
            <diffuse value1="1.0" value2="0.5" value3="0.5" value4="1"/> <!-- Again I do not know what they should be called -->
        <!-- etc -->
        </generated>
        <custom name="foobarmaterial" filename="foobar.material />
    </materials>
    <sounds>
        <sound filename="foobar.ogg" animation="dig" volume="0.8" frame="23"/>
    </sounds>
    <ai filename="foobarai.as"/>
</creaturetype>


Attributes are good

Agreed

In the end we should also provide a graphical editor for the creatures (maybe directly from the game), then the user won't have to care about the internals anyways.

Yep that would be very useful when syncing the sounds to the animations. Also I added the volume for each sound file above and changed the delay to a frame attribute that refers to the mesh animation frame where the sound should be played.

You can use attributes for whatever you want but they were created for one single purpose: meta-data.

Attributes are meant to give information about the data stored not to store data themselves, everything that is relevant should be an element. Still, nothing prevents you from putting data in attributes instead of elements.

Personally I feel that using attributes for data is wrong, but may be due to having worked with XML for quite some time and used some libraries that didn't support attributes.

In my opinion attributes can store both data and metadata, but elements should not store data them-self i.e. in the element, unless they store text or blobs of binary data, elements should only define structure the xml files.

[oln]

{l Code}: {l Select All Code}

            <specular value1="0.5" value2="0.5" value3="0.5" value4="1" value5="20"/> <!-- I did not know what to call these attributes -->
            <diffuse value1="1.0" value2="0.5" value3="0.5" value4="1"/> <!-- Again I do not know what they should be called -->


That would be red, green, blue, alpha, respectively. And for the specular component, the last value is shininess.

Attributes are good

Agreed

In the end we should also provide a graphical editor for the creatures (maybe directly from the game), then the user won't have to care about the internals anyways.

Yep that would be very useful when syncing the sounds to the animations. Also I added the volume for each sound file above and changed the delay to a frame attribute that refers to the mesh animation frame where the sound should be played.

You can use attributes for whatever you want but they were created for one single purpose: meta-data.

Attributes are meant to give information about the data stored not to store data themselves, everything that is relevant should be an element. Still, nothing prevents you from putting data in attributes instead of elements.

Personally I feel that using attributes for data is wrong, but may be due to having worked with XML for quite some time and used some libraries that didn't support attributes.

In my opinion attributes can store both data and metadata, but elements should not store data them-self i.e. in the element, unless they store text or blobs of binary data, elements should only define structure the xml files.

I don't really know what's "optimal", though from a quick google search it seems something closer to knitter's approach is more common. E.g https://www.ibm.com/developerworks/xml/ ... index.html.
Maybe not be as strict as saying attributes is for pure metadata only, but it seems that it's at least usually used for a value that is defined elsewhere, e.g in our example the type of texture and material, but not the names and filenames.

[svenskmand]

If we keep data as attributes it makes the files shorter and easier to read. Also if we have values inside of elements we would have to passe this data ourself (that is if we store more than one value inside of each element, e.g. the specular element would have its 5 values in a single element which we would then have to parse ourself) instead of letting the xml parser do it for us.

So I see no advantage at all in storing data other places than in the attributes. Remember that we implicitly store data in the structure of the elements, like it is done with the sounds element which stores sound elements inside of it.

In my opinion your link says the same as I say, i.e. use elements when it makes sense and likewise with attributes

Oln: I would like to help out making the XML Schema definition file if you need it

[Knitter]

Just use what makes sense for you , the only thing you need to consider is any limitation on the parsing engine but even that is something that shouldn't be a concern with the existing parsers.

As for the discussion about the attributes, they were created for storing meta-data, please see the W3C site or the XML group one, but I have never seen them used that way and it ended up being one of those features that were designed with on thing in mind and ended up being used in a different way.

[StefanP.MUC]

Up until now I only did easy works with XML (e.g. editing existing ones), so my previous opinion can be considered as "low priority".

As a more general argument I'd say, like Knitter said: Stick to the official specification. And if they tell us to use elements, then we should use elements, if this is the intended way and best practice.

[oln]

{l Code}: {l Select All Code}

<?xml version="1.0" encoding="UTF-8" ?>
<creaturetype class="BasicWorker">
    <creaturedata> <!-- better name for this? -->
        <name>foobar</name>
        <bed type="foobarbed">
        <!-- Could allow defining a custom bed here -->
            <scale>
                <x>1.0</x>
                <y>1.0</y>
            </scale>
        </bed>
        <scale>
            <x>1.0</x>
            <y>1.0</y>
            <z>1.0</z>
        </scale>
        <levelincrease> <!-- any suggestions for a name here? -->
            <!-- later we can add more stuff here, such as spells -->
            <hp>5</hp>
            <mana>5</mana>
        </levelincrease>
        <sightradius>15</sightradius>
        <digrate>10</digrate>
        <dancerate>5</dancerate>
        <movespeed>10</movespeed>
        <!-- faction/race stuff when that's decided on -->
    </creaturedata>
<!-- etc -->
    <model>
        <mesh>foobar.mesh</mesh>
        <materials>
            <!-- For most creatures/objects we just want a normal mapped or per pixel
            lighting material, and specify textures, and optionally lighting parameters.
            so specifying it like this might be a-->
            <generated type="normalmap_multipass" space="tangent">
                <name>foobarmaterial</name>
                <texture type="diffuse">foobar.png<texture> <!-- There can be multiple of these -->
                <texture type="normalmap">foobar_nm.png<texture> <!-- alternatively, have an element type for each texture type -->
                <specular>
                    <red>0.5</red>
                    <green>0.5</green>
                    <blue>0.5</blue>
                    <alpha>1</alpha>
                    <shininess>20</shininess>
                </specular>
                <diffuse>
                    <red>0.5</red>
                    <green>0.5</green>
                    <blue>0.5</blue>
                    <alpha>1</alpha>
                </diffuse>
            <!-- etc -->
            </generated>
            <!-- this is how materials are defined currently, with a basic material file, and the parameters for the shader generator to
            enable normal mapping.-->
            <standard type="normalmap_multipass" space="tangent">
                <name>foobarmaterial</name>
                <filename>foobar.material</filename>
                <texture type="normalmap">foobar_nm.png<texture>
            </standard>
            <!-- we might also want to let some materials use custom shaders -->
            <custom >
                <name>foobarmaterial_with_custom_shader</name>
                <filename>foobar2.material</filename>
            </custom>
        </materials>
        <sounds>
            <sound animation="dig" volume="0.8" frame="23"/>
                <filename>foobar.ogg</filename>
                <volume>0.8</volume>
                <frame>23</frame><!-- would we want to be able to specify multiple frames? -->
            </sound>
        </sounds>
    </model>
    <!--Don't think we will customise creature ais any time soon,
    but eventually we could have some different options to choose from -->
    <behaviour type="aggressive" />
    <!--
    <behaviour type="script">
        <ai type="angelscript">
            <filename>
                foobar.as
            </filename>
        </ai>
    </behaviour>
    -->
</creaturetype>


An updated suggestion, based on the more element-centered approach.

[StefanP.MUC]

Yes, now I like the element version better from readability after seeing both cases next to each other.

<creaturedata> could simply be <data> (it's clear that it's about a creature, because the parent is <creaturetype>, maybe the top level element could also simply be called <creature>).

<levelincrease> could be <levelup>.

[svenskmand]

Lets see this from a practical point of view, why store data ONLY in elements when 1) it makes the XML files allot larger, 2) makes them allot harder to edit by hand (and we will sure have to do this for a while) and 3) nobody else does it, just look at html, they use attributes and elements mixed, e.g. the a tag has a href inside, but a link is normally part of the data and not the metadata (just look at the source of this forum page). Another example is svg-files look at this svg-file made in Inkscape.

Putting all the data in elements as done in oln's last post bloats the xml file and makes it hard to hand-edit.

[StefanP.MUC]

Some of the elements can really be attributes, I assume, but the "main property" of specific data should still be an element. Small example:

{l Code}: {l Select All Code}

<creature name="Horny" type="EvilLordOfHell">
  <data>
    <bed type="ThroneOfDoom">
      <scale x="2" y="3" />
    </bed>
    <scale x="30.0" y="10.0" />
    <levelup hp="+1337" mana="+1337" />
  </data>
</creature>


[svenskmand]

In that example it is all attributes, so what do you mean that the main data is in a element?

[StefanP.MUC]

The <scale> thing for example is still an element (with atributes x and y). That's better than having only element <creature> with the atributes scalex, scaley, name, type, ... (<creature name="" sclaex="" scaley="" type="" />)

Shortly spoken: Data that belongs semantically together should have an element with the actual data as attributes (like the scale duplet).

[svenskmand]

Ahhh yeah ok, yes agreed that is a good idea. As long as we do not headless make an element for each data value it should be good. Your example is good

[StefanP.MUC]

I did some further thinking, reading and comparing to my knowledge about other languages:

Intuitivly (from the sound of the words "element" and "attribute" and from the usage in other programming languages and in science) I would expect that all raw data (numbers, strings, ...) are stored in attributes. And attributes themselves are "member variables" of an "object" (element) and the objects are grouped in categories.
So, create as much elements as needed to create a "nice-to-read" structure and store the actual raw data in attributes.
In other words: the numbers and strings themselve shouldn't create a new "level of deepness", but the grouping parent elements can and should.

So, I think this is my final opinion after reading the several opinions and articles (for a code example see my second-previous post).

[svenskmand]

Yes and this is the same as I talked about in my first post. I.e. we store the data in attributes and use the elements to give structure to the file

[svenskmand]

When I have worked with XML I have used JDOM which is DOM for Java, and their tools work excellent, so I would recommend the C++ version of this tool, which seems to be this (from a quick google) which is Apaches implementation of the DOM interface in C++.

Also Xerces is in the Ubuntu libs it is called libxerces-c* (there are many packages), so we do not need to depend on yet another PPA

[oln]

I've used DOM in C++ with Qt (and xmlpullparser when doing android development in java.), so I assume it's not all that different, since the DOM interface is quite standardised.

If we don't need xml schema support it might be easier to use http://www.grinninglizard.com/tinyxml/. Then we can use the source directly like with angelscript.

[svenskmand]

I would assume that we want the XML to be verified? Also having the XML Schema definition serves as a easy reference if people need to handedit the XML files.

But is it not nicer to depend on a external available library than to yet again copy external source code into our project and maintain it when new updates arrives?

[oln]

Since we've made build scripts for both platforms anyhow (though it would be nice with one for MSVC as well), then using an external library shouldn't be a problem. So maybe we should go with xerces.

[svenskmand]

Why should it not be possible to make a buildscript/project file for MSVC also if we use xerces? Or was that just a random though that we should have a MSVC project file also?

My vote lies with xerces.

I will try and make a quick XML Schema file, what do we need to be in it? Here is what I can think of

• Stats like: health, melee attacks, missile attacks, spell attacks, selfcast spells, bedtype,
• Graphics stuff: textures (with specular, normalmaps, etc), scaling, etc.
• Sounds: sounds the creature makes with information about which frames of the animation they should be played to and which animation they belong to (attack, walk, sleep, etc.), etc.
• Scripting: AI (if we need that for each individual creature), "scripted-actions" for use in maps, different win/goal conditions, etc.

I have surely missed something, so please extend this list.

Also we will surely have to add stuff as we go along, which also makes it easier to have a XML Schema file, as we will then get a validation exception if we try to use an XML file following and old and incompatible version of the XML Schema file.