Runtime defined systems, components, and entities (C++, ECS)

Runtime defined systems, components, and entities (C++, ECS)

Postby domtron » 28 Oct 2017, 15:39

EDIT: Seems my title got cut off so I'll put the 'tags' I had listed here: game engine, ECS, mods, c++, lua

Introduction
I'm trying to create a tech-demo for an engine based on several concepts. The engine is copying Minetest's structure for mods where the engine is itself a standalone executable and the mods are loaded/run after the engine is executed to add all game content. Mods are written in runtime compiled C++ (main focus and why it is a tech-demo to see if I can get it to work) and Lua/LuaJit. And yes, the tech-demo and possible resulting engine would be under a FOSS license. I haven't put the source up yet as it currently does next to nothing, but I can if someone wants it.

My issue
I'm trying to implement an Entity-Component-System so mods can define systems and components. However, all the ECS libraries I have looked at use C++ templates, which are compile-time only to my understanding, for adding and tracking systems and/or components. This causes an issue since the engine loads mods at runtime which define new systems and components, but since the templates are evaluated when the engine is compiled mods cannot use the ECS library's add system and/or add component functions. Templates also causes my Lua scripting language problems since I cannot use a single wrapper class as mods would overwrite each other when registering different instances of the wrapper class to the ECS library. (if that makes sense, >.< ) Then again maybe I'm just an idiot that doesn't understand templates very well.

I have looked at entityx and artimis-cpp in detail and a half dozen more superficially and they all use templates.

Reference: http://tilemapkit.com/2015/10/entity-co ... x-artemis/


So here is an example of my problem. The following code sets up a component and a system in C++ using entityx.

{l Code}: {l Select All Code}
#include "entityx.h"

struct PrinterComponent {
    std::string text;
    PrinterComponent(std::string _text) : text(_text) {}
};

struct PrinterSystem : entityx::System<PrinterSystem> {

  public:
   
    PrinterSystem() { }

    void update(entityx::EntityManager &es, entityx::EventManager &events, entityx::TimeDelta dt) override {
        es.each<PrinterComponent>([dt](entityx::Entity entity, PrinterComponent &printer_data) {
       std::cout << printer_data.text << std::endl;
        });
    }

};


Then the code must call "entity_manager.systems.add<PrinterSystem>();" to add the system. Similarly the code must call "entity.assign<PrinterComponent>("test1");" to add components to an entity. This is fine in a normal game, but with my engine I have both C++ and Lua mods that need to define systems and components during run time.

All the libraries I looked for used c++ templates. The only thing I can think of is to write my own which I really wanted to avoid since I doubt I could make something efficient enough for a game with many entities plus it would take time away from my focus (C++ Runtime Compiled mods).

Basically I'm asking if anyone knows of a decent ECS library that doesn't use templates.



Additional Notes
I'm trying to use ECS because I like the modularity and think it would work really well in a modding environment. Three different mods can define 3 different systems along with their components and then a fourth mod can mix and match them to create several different entities. I'm not really sure how I would do it without ECS.

I'm trying for C++ Runtime Compiled mods/code because I really think it is a way to make your mods 1st class citizens. I'm hoping for little to no overhead with interfacing with the engine since they are both C++ so there should be very little slowdown. Of course c++ mods will still need some sandboxing, which adds overhead, to protect users. There is also several prior attempts, though not for mods more for rapid development, so it isn't an unprecedented idea.

In my project, mods extend the engine in 4 ways: defining and redefining systems and related components (where i'm having an issue with ECS libraries), extending or overriding the engine's API, one time running of configuration API functions, and defining code not associated with a single entity that runs on a timer or the global time step.
Last edited by domtron on 28 Oct 2017, 19:51, edited 1 time in total.
domtron
 
Posts: 75
Joined: 10 Jun 2013, 16:22

Re: Runtime defined systems, components, and entities (game

Postby Lyberta » 28 Oct 2017, 17:15

domtron {l Wrote}:Mods are written in runtime compiled C++


Unless you want to embed C++ compiler such as Clang into your engine, there is no such thing as runtime-compiled C++. The classical way of implementing plugin system is via "interfaces" which are abstract classes without data members and all of their member functions are public and pure virtual. You compile your mods into dynamic libraries that export a function that returns pointer to the interface, then your engine loads the library and calls that function. After that you simply use virtual functions as usual.

I have implemented this system in my engine. I've decided from the start that my engine will be very modular and parts of it could be replaced without recompilation. So I've split game code into separate libraries and non-ISO stuff into drivers that are also libraries. The core is defined in ftz General

All interfaces are derived from Interface class and are registered with InterfaceManager. An interface that needs another interface can use InterfaceManager to get the pointer via UUID. After I've implemented this system, I've found that it is very similar to COM.

Dynamic library loading is implemented in ftz Platform.

Here's how everything is loaded:
  • The client or server executable are statically linked with filesystem and library drivers because they both are needed to load other libraries. Engine is implemented as a library.
  • At first, instances of both drivers are created.
  • Their pointers are passed to Loader which finds engine library (libftzengine.so on GNU/Linux) and loads it.
  • Executable calls CreateEngine which creates an engine instance. The engine library has no global state so you can spawn any number of engines in your process if you implement your own executable.
  • SetGameName sets the name of the game that will be loaded.
  • In case of client executable, StartClient is called. This makes engine find and load drivers required for the client. Then the engine starts client thread which loads game client library (libClient.so on GNU/Linux), creates the instance of game client and starts the client loop.
  • If client requests to start a server, StartServer is called which loads server drivers if necessary, starts server thread, load game server library (libServer.so on GNU/Linux), creates the instance of game server and starts the server loop.

The drivers that will be loaded depend on the config, user can choose which drivers they want to use. For example, I have 2 network drivers, one that uses TCP and one that uses UDP.

Engine needs to be a library so drivers and games can link to it via -l flags.

The loading logic is in library driver: interface, implementation.

As with yours, my engine does next to nothing but I'm trying to code it the right way without rewriting it a lot. So I think I got modularization right.

I plan to split game client code into menu code that will stay C++ and other code that will be WebAssembly. That way I will be able to keep using C++ but will allow servers to upload client-side mods to clients that will make total conversions possible by simply connecting to the modded server. Similar system is implemented in DarkPlaces engine.
Some crazy person on the Internet.
User avatar
Lyberta
 
Posts: 307
Joined: 19 Jun 2013, 10:45

Re: Runtime defined systems, components, and entities (game

Postby domtron » 28 Oct 2017, 19:49

Lyberta {l Wrote}:Unless you want to embed C++ compiler such as Clang into your engine, there is no such thing as runtime-compiled C++.


That is exactly what I want to do.

I have played with both Clang's dev library and something called libgccjit, but I had trouble getting examples to compile for the first and the second had issues I'm not fully remembering atm. Not enough documentation though I plan to try again later. Right now I'm using a library called RuntimeCompiledC++ which calls the external binary for GCC or Clang to compile code and then relinks it in real time using an external binary for ld or similar. The library is actually intended as a "Hot swappable code" implementation to speed up development by recompiling any changes you make seamlessly and re-initializing the data from the old class. Basically you can change a hardcoded value or add a new function call and watch in realtime as you entities change in behavior.

My use case is a little "simpler" then RuntimeCompiledC++'s in that I want to just compile mods at startup and I don't need to reinitialize data, but I also want the compiler and build stack to be integrated in the engine to avoid extra work for the users and the issue with ABI incompatibility (I think it's called? Error that happens when you try to use two chunks of code together that were compiled by different build systems). However for testing purposes RuntimeCompiledC++ is working ok. Early on, I successfully compiled a simple single file c++ "mod" that printed a string before I broke everything while building out the architecture for my engine so I know it works to some extent.

As I said in my opening post I have a bit of Prior Art to support the idea that it is possible. I can share more links as I've been collecting a number from my haphazard research.

Lyberta {l Wrote}:You compile your mods into dynamic libraries that export a function that returns pointer to the interface, then your engine loads the library and calls that function.


My issue with this for mods is that it places more work on the modders when cross platform comes up. They have to recompile their mod for each platform which they may not be willing to do. You would then end up with a bunch of mods that the users have to either compile manually or go without because their platform isn't supported. My goal is to eliminate as much work as possible while still reaping the benefits of native C++ mods.

Lyberta {l Wrote}:... but I'm trying to code it the right way without rewriting it a lot. So I think I got modularization right.

modularization is awesome isn't it. :D I'm going for two different levels. Mods which the users interact with on a regular basis and then the engine itself where each part (mod handling, graphics, entity management, etc) interacts through a single interface class so the whole thing can be replaced without too much trouble.

Your engine does look pritty cool, but it sounds like alot of work is put on the developer (need to recompile for each platform) and it might not be as end-user friendly as I'm hoping my approch could be.

I'm not sure how helpful your comments on the Interface approch will be to me. I'll think it over and see if it is usable, but I'm still hoping someone knows of a ECS that doesn't rely on C++ templates. I probebly just need to write my own. :/ Maybe using a modified version of your approch.

Thanks for the response and info! :)
domtron
 
Posts: 75
Joined: 10 Jun 2013, 16:22

Re: Runtime defined systems, components, and entities (game

Postby Lyberta » 30 Oct 2017, 16:51

domtron {l Wrote}:My issue with this for mods is that it places more work on the modders when cross platform comes up. They have to recompile their mod for each platform which they may not be willing to do. You would then end up with a bunch of mods that the users have to either compile manually or go without because their platform isn't supported.


I "solved" it by choosing AGPL. So the source code of all mods on public servers will be available. I doubt someone will host 24/7 server not on GNU/Linux.

Also, I'd rather compile my mods in the IDE that integrates with compiler and other tools such as static analysis rather than messing with engine. I had the same problem writing Lua. You can't know if your script syntax is correct until you load it into the game. I really hate it.
Some crazy person on the Internet.
User avatar
Lyberta
 
Posts: 307
Joined: 19 Jun 2013, 10:45

Re: Runtime defined systems, components, and entities (game

Postby dulsi » 31 Oct 2017, 15:41

domtron {l Wrote}:My issue with this for mods is that it places more work on the modders when cross platform comes up. They have to recompile their mod for each platform which they may not be willing to do. You would then end up with a bunch of mods that the users have to either compile manually or go without because their platform isn't supported.

I do think that not requiring a compile is nicer for modders and make redistribution easier. However, if you are spawning a regular C++ compiler, I don't think you are really going to prevent mods that are not cross platform. To ensure cross platform, you would need to restrict them from including system headers and doing system dependant code.

Lyberta {l Wrote}:Also, I'd rather compile my mods in the IDE that integrates with compiler and other tools such as static analysis rather than messing with engine. I had the same problem writing Lua. You can't know if your script syntax is correct until you load it into the game. I really hate it.

Well you know it compiles. Not quite the same as works but I understand your point. I prefer the addition checks of compiling but for modding using a simplier language can encourage people to try. (I don't necessarily think you will get better mods. The people writing really good mods would probably do so in any language. My youngest has enjoyed adding blocks to minetest. They aren't generated in the world. They are oddly colored but he enjoys being about to do that.)
dulsi
 
Posts: 75
Joined: 18 Feb 2016, 15:24

Re: Runtime defined systems, components, and entities (game

Postby Lyberta » 01 Nov 2017, 01:19

dulsi {l Wrote}:Well you know it compiles. Not quite the same as works but I understand your point.


I tend to do some typos here and there and in IDE I just hit F7 and compiler easily tells them. Now in Lua I have to sometimes resetup the game, wait for it to load and then give me error. This is so much slower.

In modern C++ if it compiles, it most likely gonna work. Modern C++ solves many bugs from C and old C++ at the design level. Not quite Rust but still good.
Some crazy person on the Internet.
User avatar
Lyberta
 
Posts: 307
Joined: 19 Jun 2013, 10:45

Re: Runtime defined systems, components, and entities (game

Postby domtron » 05 Nov 2017, 02:36

Lyberta {l Wrote}:Also, I'd rather compile my mods in the IDE that integrates with compiler and other tools such as static analysis rather than messing with engine. I had the same problem writing Lua. You can't know if your script syntax is correct until you load it into the game. I really hate it.


I don't see how that effects using C++ mods... You just develop it in your IDE and compile/run in the engine. As you write the code you get all the pre-compile benefits from the IDE like typos correction and the like. And a number of IDE's allow you to configure what commands the run and compile button/action run so you could just replace the normal operation with one that calls the engine. I certainly wouldn't be opposed to adding a dry run argument to the engine that just runs the ModFramework portion so it doesn't take up unneeded resources spinning up the graphics, audio, environment, etc, etc.

Not sure about the runtime tools like "static analysis" as I don't use them. I'm personally very minimal when coding. I use gedit, a terminal to run the compile/linking commands (usually through make or cmake), and occasionally break out GDB or valgrind if I run into a sticky problem. Anyway I imagine you could configure the IDE tools to watch the engine process or maybe something can be done on the engine side to expose only your code to the tools. Thought that falls under advanced features if I can even get the tech demo functional.

dulsi {l Wrote}:However, if you are spawning a regular C++ compiler, I don't think you are really going to prevent mods that are not cross platform. To ensure cross platform, you would need to restrict them from including system headers and doing system dependant code.


I imagine I need to do that anyway for sandboxing purposes. If you can access any system level STL or library a malicious mod could cause a lot of damage. My thoughts on that is to have an Engine Module dedicated to interfacing with any platform specific code and exposing them to mods as a portion of the API. That both enables cross platform code and basic security. The other is to only allow approved C++ libraries in the official mod repository. (I have a sort of rough planned out community for the engine as well including mod distribution)

----------------------------------------------------------------------------------

Back on topic, I implemented a rough ECS that seems to fit my needs. On the one hand it is another thing I have to implement and maintain, on the other it gives more control.

Basically I have an ECS manager that allows registering a pointer to a child of my base System class. So any class that inherits from that class can be registered, it just has to implement initialize and update entity functions. IT is very simple right now and I'm sure there are many problems that will crop up. D: But it is working so far so that is good.

Below is the example code for using it:

{l Code}: {l Select All Code}
struct PrinterComponent : EntityComponent {
    std::string text;

    PrinterComponent(std::string _text) : text(_text) {}
};

class PrinterSystem : public EntitySystem {

  public:

    void inilizeEntity(Entity* ent){
       PrinterComponent* comp = new PrinterComponent( "test"+std::to_string(printer_count) );
       printer_count++;

       //call the data and save the resulting object pointer
       ent->components.insert( {"PrinterComponent", comp} );
    }

    void updateEntity(Entity* ent) {
        PrinterComponent* printer_data =
           dynamic_cast<PrinterComponent*> (ent->components["PrinterComponent"]);
        std::cout << printer_data->text << std::endl;
    }

  private:
    int printer_count = 0;
};

//adding the above to the ECS manager
entity_manager.registerSystem("PrinterSystem", new PrinterSystem());

//create some test Entities and add the system to it
EntityID entity = entity_manager.createEntity();
entity_manager.addSystemToEntity(entity, "PrinterSystem");

entity = entity_manager.createEntity();
entity_manager.addSystemToEntity(entity, "PrinterSystem");


Biggest problem right now (that I can see) is passing data to the individual entities. I'm going to have to look into a way to arbitrarily pass data through the "addSystemToEntity" function.

For now I'm working on implementing the Lua wrapper since my Lua mods are the only ones currently working and I'd like to see if it actually solves that part of the problem. If that works out, I'll probably implement an ASCII graphical interface first (I love "ASCII" games) so the engine will have all parts implemented in some form and be somewhat usable. Then I'll hunker down and focus on the C++ mods and rounding out the other engine modules.

Thanks for the responses. :)
domtron
 
Posts: 75
Joined: 10 Jun 2013, 16:22

Re: Runtime defined systems, components, and entities (game

Postby Lyberta » 06 Nov 2017, 15:47

domtron {l Wrote}:You just develop it in your IDE and compile/run in the engine.


Except it is much easier to compile without running the engine because it better integrates with the IDE.

domtron {l Wrote}:As you write the code you get all the pre-compile benefits from the IDE like typos correction and the like.


Actually, a lot of functionality is based on partial compilation in order to get data about types.

domtron {l Wrote}:And a number of IDE's allow you to configure what commands the run and compile button/action run so you could just replace the normal operation with one that calls the engine.


Yes, except they are much harder to configure.

domtron {l Wrote}:Not sure about the runtime tools like "static analysis" as I don't use them.


They help to find a lot of bugs.

domtron {l Wrote}:If you can access any system level STL or library a malicious mod could cause a lot of damage.


That matters only to client-side code that is automatically uploaded by the server. There is no point in sandboxing server-side code which is installed manually because you will simply hurt the moddability.

domtron {l Wrote}:
{l Code}: {l Select All Code}
PrinterComponent* comp = new PrinterComponent( "test"+std::to_string(printer_count) );


Avoid naked new.

domtron {l Wrote}:
{l Code}: {l Select All Code}
ent->components.insert( {"PrinterComponent", comp} );


You didn't check for nullptr.

domtron {l Wrote}:
{l Code}: {l Select All Code}
PrinterComponent* printer_data = dynamic_cast<PrinterComponent*> (ent->components["PrinterComponent"]);


If that calls std::map::operator[] then you are shooting yourself in the foot.
Some crazy person on the Internet.
User avatar
Lyberta
 
Posts: 307
Joined: 19 Jun 2013, 10:45

Re: Runtime defined systems, components, and entities (game

Postby GunChleoc » 07 Nov 2017, 22:35

Lyberta {l Wrote}:If that calls std::map::operator[] then you are shooting yourself in the foot.

That's a good one, thanks for the link :)
User avatar
GunChleoc
 
Posts: 271
Joined: 20 Sep 2012, 22:45

Who is online

Users browsing this forum: No registered users and 1 guest