Move to XML, XML Parsing,implementation of Pipes w/h threads

Move to XML, XML Parsing,implementation of Pipes w/h threads

Postby paul424 » 20 Mar 2014, 20:10

Bertram is a titan of work , he works for a few dozens of paul424's :D !
I am moving the game data to XML format.
Well what is XML http://en.wikipedia.org/wiki/XML everyone knows.
Using it should have 3 advantages :

    Self-Description of the used data ( no misterious byte sequences " it might be health, it might be color of eyes ;"
    External checking of map file validity -- if say xmllinit validates files against proper XSD http://en.wikipedia.org/wiki/Xsd template, than the OD program won't crash
    Composability and data reuse -- using the xs::include directive we can insert one file definitions into another ( similar to c++ include but smarter )


What needs to be done ?
    Transition from level/Tile.level to the bunch of XML's files and their XSD templates ( OD_path/XML [ ALREADY DONE] )
    Writing the proper XML parser , which would read the data.


I want to use libxml++ , which handles two common methods of XML parsing : http://libxmlplusplus.sourceforge.net/docs/manual/html/ar01s02.html#id2504579 : DOM and SAX plus one minor TextReader . I can use either SAX or TextReader -- those are two methods which use bottom-up parsing approach .

SAX looks more nice , plus it is possible to parse in chunks, that is send data to parser and ask to parse it a little instead of sending the whole file at once .
For now from the point of view of XML , game data is one big entity GameMap, which is splitted across number of files , due to magic of xi:include directive :
{l Code}: {l Select All Code}
<?xml version="1.0" encoding="UTF-8"?>
<GameMap xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:noNamespaceSchemaLocation="file:/home/tom/Opendungeons/opendungeons/XML/xsd/MapDefinitionSchema.xsd"
 xmlns:xi="http://www.w3.org/2001/XInclude">
    <MapSizeX>
        400
    </MapSizeX>
    <MapSizeY>
        400
    </MapSizeY>
    <xi:include href="../Seats/SeatList.xml"/>
    <xi:include href="../Goals/GoalList.xml"/>
    <xi:include href="../TileContainers/TileContainerList.xml"/>
    <xi:include href="../Rooms/RoomList.xml"/>
    <xi:include href="../Traps/TrapList.xml"/>   
    <xi:include href="../Lights/LightList.xml"/>
    <xi:include href="../CreatureDefinitions/CreatureDefinitionList.xml"/>   
    <xi:include href="../Actors/ActorList.xml"/>
   
</GameMap>


Problem is we need to write our own procedure for unfolding the xi:include directives. For now the fullty unfolded GameMap file would size about 1MB .
Also the bottleneck is the file seek on disc.
Instead of waiting endlessly for both file read and than parsing we can send each small chunk we already have to the parsing .
The most common pattern for that is Pipes, I have had to implent Pipe templates which would be similar to Unix pipes , but use c++11 std lib facilities ; I have already come up with template hierarchy , there is even small demo ... http://pastebin.com/stVLyeUd it works pretty nice , though you would like some syntatic sugar when setting pipes instead of

{l Code}: {l Select All Code}
    NumberSource<long> localSource;
    PipeDouble<long> localPipe;
    localSource.setSink(&localPipe);
    OutputSink<long> localSink;   
    localPipe.setSink(&localSink);


That's what would help us processing XML xs:include
More later....
Bertram and others : Ask more what do you want to know about that :D .
User avatar
paul424
OD Moderator
 
Posts: 660
Joined: 24 Jan 2012, 13:54

Re: Move to XML, XML Parsing,implementation of Pipes w/h thr

Postby paul424 » 22 Mar 2014, 14:41

On the other hand , faichele mentions to try using the ID way approach, though desiccions we have ...
User avatar
paul424
OD Moderator
 
Posts: 660
Joined: 24 Jan 2012, 13:54

Re: Move to XML, XML Parsing,implementation of Pipes w/h thr

Postby Bertram » 25 Mar 2014, 11:06

@faichele: Could you describe more the ID approach, so we can have an idea?

@paul: The XML parser is it's own is a good idea. I'll have to look at the actual code to tell though. :)
A thing that is bothering me is that you're using 8 sub-folders and 9 files to describe only one map. (?)

I'm fine with splitting the creature definitions from the rest at already decided in the issue tracker and the forums, but could you try simplifying the map data before me move further?

Best regards,
User avatar
Bertram
VT Moderator
 
Posts: 1652
Joined: 09 Nov 2012, 12:26

Re: Move to XML, XML Parsing,implementation of Pipes w/h thr

Postby faichele » 01 May 2014, 11:18

Hello!
Bertram {l Wrote}:@faichele: Could you describe more the ID approach, so we can have an idea?


See the following example.
Instead of including different XML files into a map definition, the parsing is done on start-up for all definitions (creatures, tile types, other game objects), and only map definitions are parsed when a level is loaded.
This makes the parser implementation simpler: No need to resolve definition dependencies with multiple parsing passes. The parsing order is "fixed" in code, determined by the dependencies on different game object types that need to be defined before others can be loaded (e. g. faction info before creature definitions).
Also, this results in a better "moddability": Parsing for game objects isn't necessarily restricted to a single file per game object type, but can be distributed over several XML files, which modders could use to modify existing definitions, or to add new creature types.

{l Code}: {l Select All Code}
<CreatureDefinition>
        <CreatureType>CREATURE_BIG_KNIGHT</CreatureType>
   <CreatureTypeName>Big Knight</CreatureTypeName>
        <CreatureStats>
            <Level>6</Level>
            <Health>5</Health>
            <Mana>5</Mana>
            <Wager>50</Wager>
       <MoveSpeed>1.2</MoveSpeed>
       <SightRadius>15</SightRadius>
   </CreatureStats>
   <CreatureCombatStats>
       <AttackStats>
      <Attack>
        <CombatType>COMBAT_MELEE</CombatType>
        <Level>6</Level>
        <Damage>25.0</Damage>
      </Attack>
            </AttackStats>
            <DefenseStats>
                <Defense>
        <CombatType>COMBAT_MELEE</CombatType>
        <Level>5</Level>
        <DamageReduction>50.0</DamageReduction>
      </Defense>
      <Defense>
        <CombatType>COMBAT_RANGED</CombatType>
        <Level>4</Level>
        <DamageReduction>20.0</DamageReduction>
      </Defense>
      <Defense>
        <CombatType>COMBAT_MAGIC</CombatType>
        <Level>3</Level>
        <DamageReduction>10.0</DamageReduction>
      </Defense>
            </DefenseStats>
        </CreatureCombatStats>
   <CreatureAIType>AI_STRONG_FIGHTER</CreatureAIType>
   <CreatureArtDefinition>
       <MeshName>Knight.mesh</MeshName>
       <RigName>Knight.skeleton</RigName>
       <Material>
      <Name>KnightMaterial0</Name>
      <TextureMap>
          <normalmap filename="filename0" space="tangent"/>
          <specularmap filename="filename1"/>
      </TextureMap>
      <Name>name0</Name>
      <FileName>filename2</FileName>
       </Material>
       <Scale>
         <xscale>1.2</xscale>
         <yscale>1.2</yscale>
         <zscale>1.2</zscale>
       </Scale>
       <BedMeshName>FORGE_OBJECT</BedMeshName>
       <BedDimX>3</BedDimX>
       <BedDimY>3</BedDimY>
       <SoundList>
         <Sound>
        <filename>filename3</filename>
        <animation>animation0</animation>
        <frame>0</frame>
        <volume>0</volume>
         </Sound>
       </SoundList>
   </CreatureArtDefinition>
   <CreatureSocialStats>
     <AvailableFactions>
       <Faction>FACTION_HUMANS</Faction>
     </AvailableFactions>
     <DanceRate>0</DanceRate>
     <SocialCoefficients>
         <cHumans>1</cHumans>
         <cCorpars>0</cCorpars>
         <cUndead>0</cUndead>
         <cConstructs>0</cConstructs>
         <cDenizens>0</cDenizens>
         <cAltruism>1</cAltruism>
         <cOrder>1</cOrder>
         <cPeace>1</cPeace>
     </SocialCoefficients>
   </CreatureSocialStats>
   <Scripts>
       <FileName>filename4</FileName>
   </Scripts>
</CreatureDefinition>

<?xml version="1.0" encoding="UTF-8"?>
<GameMap>
    <MapSize>
        <SizeX>400</SizeX>
   <SizeY>400</SizeX>
    </MapSize>
    <Seats>...</Seats>
    <Goals>...</Goals>

    <MapTiles>
      <TileList>
   <Type>TILE_SOLID_ROCK</Type>
   <CoordinatesX>0 3 5 ...</CoordinatesX>
   <CoordinatesY>0 2 4 ...</CoordinatesY>
      <TileList>
      <TileList>
   <Type>TILE_EARTH</Type>
   <CoordinatesX>1 3 8 ...</CoordinatesX>
   <CoordinatesY>0 3 5 ...</CoordinatesY>
   <Occupation>
     <Faction>FACTION_HUMANS</Faction>
     <CoordinatesX>1 3 8 ...</CoordinatesX>
     <CoordinatesY>0 3 5 ...</CoordinatesY>
   </Occupation>
      <TileList>
    </MapTiles>
    <MapRooms>
      <RoomList>
   <Type>ROOM_QUARTERS</Type>
   <CoordinatesX>1 1 1 ...</CoordinatesX>
   <CoordinatesY>1 2 3 ...</CoordinatesY>
      </RoomList>
    </MapRooms>

    <Creatures>
      <CreatureList>
    <Type>CREATURE_BIG_KNIGHT</Type>
    <Faction>FACTION_HUMANS</Type>
    <CoordinatesX>2 4</CoordinatesX>
    <CoordinatesY>5 5</CoordinatesY>
      </CreatureList>
    </Creatures>
    <Traps>
      <TrapList>...</TrapList>
    </Traps>
   
</GameMap>


Of course, this is just "my two cents" on the topic. Constructive criticism is welcome!

With best regards,
Fabian Aichele
faichele
 
Posts: 1
Joined: 01 May 2014, 10:52

Re: Move to XML, XML Parsing,implementation of Pipes w/h thr

Postby Elvano » 01 May 2014, 14:45

paul424 {l Wrote}:<GameMap xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="file:/home/tom/Opendungeons/opendungeons/XML/xsd/MapDefinitionSchema.xsd"
xmlns:xi="http://www.w3.org/2001/XInclude">

As I have mentioned before, you really shouldn't use absolute referencing to schemas.
Use relative references instead. That way you can't forget to change them to relative when you load them to the svn.
Elvano~
User avatar
Elvano
 
Posts: 121
Joined: 23 Sep 2013, 12:42

Re: Move to XML, XML Parsing,implementation of Pipes w/h thr

Postby Bertram » 14 May 2014, 17:11

Hi,

No I disagree with both paul's and faichele's approaches, but only partially with faichele's one.

faichele is proposing to load the creatures definition only once, which for 90% of the case is fine, of course. But the current approach (the one already in the box) is better
as level files can point to a given creatures.def file already when loading a map, permitting to change the creature definition for each level, when desired.
Meaning we can change the look of them, or the type of creature you get depending on the map, which is cool, IMHO.

Distribute data over several files is fine as long as data belonging to the same set is kept together, or you'll keep having head aches with preserving data integrity.
Separating every definition in its own file is not relevant, for instance, as it doesn't make it better for the modder, only worse, and break easily.

I do think the current separation is fine and I see no point (Go ahead, try to prove me wrong...) in separating data further.
Note that there is one thing to change though, the spawn pool of the teams will have to be declared for each, but still belonging to the level file, IMO.

Now as for Paul's approach, moving each and every level creatures, lights, rooms, traps, ... in a different file is completely insane, IMO. A level being the addition of more than 10 files each in something like 8 folders? Seriously?
Plus, the parser he made isn't reusable and already over-complicated, IMHO.

If moving to XML means one of the two approaches, I'm seriously against. I do think a simple parser should be more than sufficient for the one or two files a level is representing.

I might sound harsh, but it has to be told, guys.

Regards,
User avatar
Bertram
VT Moderator
 
Posts: 1652
Joined: 09 Nov 2012, 12:26

Re: Move to XML, XML Parsing,implementation of Pipes w/h thr

Postby Bertram » 27 Sep 2014, 12:43

Topic obsolete. Can someone unstickify this one?

Thanks!
User avatar
Bertram
VT Moderator
 
Posts: 1652
Joined: 09 Nov 2012, 12:26

Re: Move to XML, XML Parsing,implementation of Pipes w/h thr

Postby Danimal » 27 Sep 2014, 12:48

made non-sticky
User avatar
Danimal
OD Moderator
 
Posts: 1407
Joined: 23 Nov 2010, 13:50

Who is online

Users browsing this forum: No registered users and 1 guest

cron