Splitting the engine

[sternstaub]

On 05/02/2023 at 3:13 PM, Stan` said:

And obviously on some linuxes it can take months for 0 A.D. to finally arrive in Repos.

we are lucky to have the option of flatpak  (or to change linux distro; 0ad worth it... i thinkkk...) 

 

This makes me wonder: was 0ad and pyrogenesis not supposed to be seperated one day? this could ease building process, right ? 

[Stan`]

7 hours ago, sternstaub said:

This makes me wonder: was 0ad and pyrogenesis not supposed to be seperated one day? this could ease building process, right ? 

We're working on it. @trompetin17 is working on making the Editor use the game's GUI engine removing the need for WXWidgets, and thus making the editor extremely flexible. By moving all the dependencies to the mod mod, we should be able to extract the MVP from the game.

However that solves nothing. For mod versions are linked to engine versions and most mods need all the components from the game's mod. 

[sternstaub]

1 hour ago, Stan` said:

However that solves nothing. For mod versions are linked to engine versions and most mods need all the components from the game's mod. 

The game seems to have multiple workflow layers. pyrogenesis -> 0ad -> mods
There is pyro engine with the simulation and basic data handling, which would be the layer for I/O optimization. Abstract stuff in here. I think it should ideally reach a somewhat stable state as the first? So that the stability can build up to the other layers. Because better simulation and performance stability = better community games = more activity on community layer (the mods for 0ad)
Then, there is 0AD itself; it is much about the meshes, soundtracks, civ design, building sizes
(if a space game would use pyrogenesis, it maybe had very different buildings, so that would be handled by the 0ad layer)
, but also gives the basic, persistent balancing for every basic unit type. So, the game mod is kind of a.. consensual game state, which can be assumed to provide certain functionality, meshes and compatibilities. It is a common package for a community of players. In that perspective, it is more modular than the engine (generic operations and script handling, graphics rendering...), but less modular than the actual mods. That way, it allows for a greater number of players to have trust in a common gamestate and prevent OOS. Moddable game is great, but this is of course a Problem to keep track of all the versions and ensure same game behaviour between different players.
p.s., i am aware that there is a "0ad" base mod within the game, but maybe it would be helpful to handle it in different layers? Engine layer should have long update cycle, game layer with medium update cycle (seasons+balancing??), and then the community-modding-layer, which is technically bleeding-edge. But there can also be some kind of release cycles, like already done. I like to compare it to arch linux and manjaro. 
In that way, that many people can have an better access to the grassroot kind of development (not only code!), if the changes can be locally limited. Because the fundamentals on which their modding relies (the game) provide a long-term stability. Maybe it even makes sense to have different mod formats, like .pyromap, .pyrociv etc., which can then have a stable dependency system implemented (theoretically).
The Community mod is a great example in which this strength of the community-driven game is already realized! It says on the description that bleeding-edge changes are quickly added and removed. It is already a good, kind of real-time community participation (though the community might still become bigger :p and it sure will, as soon as we scaled it somehow in some ways)

//EDIT4: Also consider that the different layers would have different pools of contributors. The 0ad-game can take contributions from the modding layer. Basically like a filter. But on the modular layers, we probably see the most historians, musicians, writers, translators... stuff like that. The pyro engine as the most "static" layer would take contributions mainly from devs. Not so much music in the const arrays. But (ideally, in future) these contributions could extend to other RTS who use the engine. 

 

Still in the learning of all the C++ atm. Begging pardon for maybe stupid assumptions or questions. It is not meaned to say that the engine is "too bad" or just complaining. Much effort has already be done, and probably, if we compare engine optimizations, it is much less work than all the meshes, models, voices, music, scripting, balancing, moderation, legalese etc.
I just hope that more problems can be solved ! So much has already been achieved, and it would be very lovely to see it scale up to epic battles which can take the players back in history 

  

1 hour ago, Stan` said:

working on making the Editor use the game's GUI engine removing the need for WXWidgets

is this about the atlas editor? Concerning the files that are created by modders: maybe it is worth a thought to create user friendly toolkits for packaging mods on the long run. So that there would always be checks if the basic file stucture is correct etc. Can reduce the need to re-upload a mod and start another version for some fix of missing or wrongly placed files. When there is many people modding, then this can take some work away from the team, i imagine.
//EDIT2: i am aware that there is already a command line argument to package a .pyromod. So you are most likely already aware of that fact, considering the depth of organization on wildfire games.

Wait, what is the topic called? Naming poll... this is kind of off-topic. <\talk>

Edited February 11, 2023 by sternstaub

[Stan`]

9 hours ago, sternstaub said:

There is pyro engine with the simulation and basic data handling, which would be the layer for I/O optimization. Abstract stuff in here. I think it should ideally reach a somewhat stable state as the first? So that the stability can build up to the other layers. Because better simulation and performance stability = better community games = more activity on community layer (the mods for 0ad)

Yeah but it keeps evolving (Although most API updates break the JS part) because of performance issues the API might change.

9 hours ago, sternstaub said:

(if a space game would use pyrogenesis, it maybe had very different buildings, so that would be handled by the 0ad layer)

Working on a mod for it.

9 hours ago, sternstaub said:

p.s., i am aware that there is a "0ad" base mod within the game, but maybe it would be helpful to handle it in different layers? Engine layer should have long update cycle, game layer with medium update cycle (seasons+balancing??), and then the community-modding-layer, which is technically bleeding-edge. But there can also be some kind of release cycles, like already done

The problem is that because it's a P2P model you always need everyone to have the exact same version. Else you're gonna quickly create a split in the community.

Ideally at some point empires_ascendant (the civs contained in the public mod (the 0ad mod) should be available on mod.io. 

But with many mods comes a lot of problems because it's not user friendly to enable them.

https://trac.wildfiregames.com/ticket/5366

I believe the biggest issue would be the amount of contributors. Having the engine split from the game is a nice idea, but most mods are around the game anyway and can use .DELETED files to remove what they don't need, and we don't have that many devs so splitting the two would make dependencies hell.

9 hours ago, sternstaub said:

is this about the atlas editor

Correct.

9 hours ago, sternstaub said:

So that there would always be checks if the basic file stucture is correct etc. Can reduce the need to re-upload a mod and start another version for some fix of missing or wrongly placed files. W

People need to read the docs for that. And there are scripts. We also can't ensure everything is rightly placed cause one has no idea what a mod will do.

9 hours ago, sternstaub said:

EDIT2: i am aware that there is already a command line argument to package a .pyromod. So you are most likely already aware of that fact, considering the depth of organization on wildfire games.

This is the only correct way to do it (and that's what the github action does) else the game is gonna have to cache the assets on the first run.

 

[sternstaub]

Note: Perhaps it could even be worth a thought to have four components to the game:

1) pyro engine - game logic. to be honest, it would be very cool if it had it's own .pyroscript language / interpreter like TES games (very mod friendly), but that is a lot of work. still, i like the thought.
2) passive pyro libs - libraries with all the models, textures etc. which can be used by pyro engine computation. those do not compute anything themselves. We might call them "passive objects".
The libs should not only contain 0ad base-game-content, but also content from mods, so there would eventually be a large library to use for extending the game. This means: all players have the pyrolib available (dependency of ->3), although the base game (->3) does not implement all of it's content. Anyway, mod resources could be available to easily use or derivate from, independent of the mod state. I think about 1000ad for example, which has a lot of buildings / models. Or delanda est. 
Also consider that art designers can use the existing models from the libs as blueprints / starting points; slightly alter a certain resource, creating a new one in the process, which can then be added to the libs again, etc.
You may add as many libs as you wanted, since they do not do anything on their own, as long as they don't overwrite files from each other.
Also, i do not want to imply that there would only be ONE pyro-lib. The player could use many libs, as long as they do not conflict with each other. But my intention is for the player to only need ONE library to have all the basic resources needed by many mods, creating a large compatibility layer, so to speak. Smaller, player-created libs should be merged into the master lib as soon as they are in a fine state. (Questions... how to imagine a release cycle here? Many updates? Less updates? how to update? User-friendy method possible? auto-update? tedious clicking?)
3) "master file" (for instance 0ad, 1000ad...) - basically an specific implementation of game logic with the libs. Consider Bethesdas .esp and .esm formats. this ->3 can be a "master file" which can be mounted, and have the mods (->4) override sections of it. These overrides should be traceable for multiplayer purpose, and it shoud be determined whether they are cosmetical or game-changing. Using a different master file would mean a big lore change, like the council of modders projects (1000ad...). Those master files would per nature have a big impact on the game itself, registering civilizations, setting tech trees and determining the game experience as such. It might also be the place for javaScript ? still very unsure about the whole scripting thing.
4) active mods - active mods would be able to "mount" the pyro-libs (->2) and connect them with game logic (->1). both 1 and 2 are hence providing the "frame" for mods. they would serve as an interface to ensure expected, predictable behaviour of modded game clients.
For instance, everyone has the 1000ad menu sprites and textures in their lib, but they load the game with "0ad master file" (->3)  (let's now rename it to "0ad-active", to clarify), so when they start the game, the engine would look into the active master file ->3 for the menus.
Another possibility is that the "master files" are only for "unit registration" (which units exist) and balancing. So that the damage, life, speed, gathering rates... of the different units are fixed by master file (->3) and cannot be changed by mod file (->4). The menus, however, would also be provided by master file (->3), but we can chose that they can be overwritten by mods. Strictly speaking, the master file -> would pose a fallback option for everything which is not overwritten by mods ->3 .
It also might or might not make sense to categorize mods, like it is done on nexusmods.com. So that a certain kind of mod would only be eglible to affect the corresponding part of the game. (shiny -> UI mod; )
For multiplayer, maybe have this requirement: player must have the same master file (->3), and they must not apply different sets of mods (->4) in the categories gameplay, corerender (visibility of units to the graphic renderer)(fictional example), technologies and some others which would have to be determined accordingly.

 

On 12/02/2023 at 12:12 AM, Stan` said:

Yeah but it keeps evolving (Although most API updates break the JS part) because of performance issues the API might change.

Are You referring to a design change in pyro API ?

On 12/02/2023 at 12:12 AM, Stan` said:

Ideally at some point empires_ascendant (the civs contained in the public mod (the 0ad mod) should be available on mod.io. 

But with many mods comes a lot of problems because it's not user friendly to enable them.

A kind of meta-library, as mentioned above, might part of a solution here? 

On 12/02/2023 at 12:12 AM, Stan` said:

but most mods are around the game anyway and can use .DELETED files to remove what they don't need, and we don't have that many devs so splitting the two would make dependencies hell.

How many active devs are working on the game (round about)? Cannot estimate.

This might be a matter of how it is done, and of the point in time. At the moment, there is the public mod for some (or most?) of the specific implementations for 0ad, if my - currently vague - understanding is correct.

What if the 0ad-implementation (->3) were to be a mod itself, but in the means of (->4) a pure game logic instruction mod? I know that we already have the 0ad base mod (/mods/public), but currently, it contains all the sounds, the maps, the unit models, materials etc.; These resources might be stripped from game logic. The resources may then be added to the public library (->2). The game logic be added to another "base mod", implementing 
At this point please consider that the game 0ad itself *is* actually an implementation of pyro engine logic. At the moment, they are entangled in source, so to speak. My proposal would thus be to generalize pyrogenesis logic, so that separation of both can be done in a practical way, and in the best case even provide a custom scripting interpreter with all the basic instructions to be used by mods. A script language tailored to the needs of RTS would be sandboxing per design. Those may be attached to units, like actor.ATTACK(attackable x), actor.BUILD(), or to the world, like placeActor(actor, position), placeActor(actor, position_anchor) or whatever. 

This is still quite vague. Several different thoughts in a combination. 

On 12/02/2023 at 12:12 AM, Stan` said:

But with many mods comes a lot of problems because it's not user friendly to enable them.

it could be eased with libraries which "stay in the background". at the moment, we have this "multiplayer friendly" option. This points to the direction of the library logic mentioned above.

On 12/02/2023 at 12:12 AM, Stan` said:

We also can't ensure everything is rightly placed cause one has no idea what a mod will do.

But the basic functionalities which are provided to mods might be pre-defined by the engine. Like handles. That basically would be a custom script language?
The process of implementing such handles - or, call them script commands - would require close communication with the modders, so that they can fit the different use cases of the mod devs. When many new functionalities are demanded by current modding projects, this might indeed imply a cumbersome update procedure. But: Once this reached a state where most demands are met, only smaller updates would be neccessary - ideally.

As soon as such a thing would be established, it would put the behaviour of mods into a more predictable frame, at the cost of mod dynamic. It also implies a close connection between modding and coreDev workflows. At this point, think of the community mod, which brings change on a much quicker pace than 0ad -> different release cycles.

Please do not take this post too serious; still need to get into JavaScript much more. Many lose thoughts, anyway.

Edited February 15, 2023 by sternstaub

[hyperion]

9 hours ago, sternstaub said:

1) pyro engine - game logic.

The game logic isn't meant to be part of the engine, sure the engine sets some limits as to what a game can do and the engine can offer some routines for some performance critical aspect to be handed off as well.

As for 0ad aka public, sure splitting off or splitting the mod itself (a la 2-4) might make sense if there where a better dependency/mod loader, see the mess the community mod created.

The all as a block situation isn't ideal for people to get started.

[Stan`]

3 hours ago, hyperion said:

The all as a block situation isn't ideal for people to get started.

It is not ideal for people to get started from scratch, yes. But it is enough for all the modders we have had in the past 11 years. Most of them don't want to reinvent a RTS.

For reference on how having a bare bone mod as first screen can affect players https://video.fosdem.org/2020/K.3.201/gamedev_spring_steam.mp4

That being said I think the approach we're taking with @trompetin17 on GitHub is the right one. One mod mod with a basic map editor, and a mod selector. All the needed shaders, minimum components and other xml files, and GUI to make something.

https://github.com/justjuangui/shouldbeinmod_pyrogenesis

All the other mods can build on top of that.

 

[hyperion]

10 hours ago, Stan&#x60; said:

 https://video.fosdem.org/2020/K.3.201/gamedev_spring_steam.mp4

Not just us but other to have trouble making a decent mod interface as well

Anyway, I had in mind things like loading the correct mods on joining a game / starting a replay, fetching mods or fetching content from host on demand etc. All those perks I see as a requirement for splitting mods as one pleases. That isn't trivial and also touches on the topic of security.

Having the editor mod only depend on mod mod is probably the right choice.

[Stan`]

14 hours ago, hyperion said:

Anyway, I had in mind things like loading the correct mods on joining a game / starting a replay, fetching mods or fetching content from host on demand etc. All those perks I see as a requirement for splitting mods as one pleases. That isn't trivial and also touches on the topic of security.

Yeah that'd be nice but security is hard to ensure. We could make sure you only can download signed mods (with an opt-in option for insecure ones but then anymod can force that to be off)  But you still have to agree with mod.io terms to download them

15 hours ago, hyperion said:

Having the editor mod only depend on mod mod is probably the right choice.

In my head base functionality is modmod. Public extends it with extra components, shaders, assets, etc. 

Also removes most of the hardcodings needed in AtlasUI, wxwidgets dependency, so it's a win. But, one has to implement more GUI features for Wxwidgets is more complete.