[Split] Mod Manager -> Mod Downloader + Mod Configurator (Modification, Addons & Maps)

[Radagast.]

History:

Personally I favour working on our Hybrid AI. Though I believe this Content Manager is essential for all our efforts.

So I wish to declare this a priority if it's fine for everyone and try to figure at least a workaround for the CoM, if lucky it may fit into the engine, but my humble self can't give any guarantuees.

Might take some time until the development takes up full momentum.

Also I don't want to hinder any other dev to pick up the task or work out something more official. There are so many things to make happen. So it's possible we - the CoM - only may achieve a workaround due to our limited knowledge in comparison to experienced pyrogenesis devs, so we might not find a permanent solution eventually.

Time will tell. Let's see what we come up with.

---

Overview
=> Mod Manager project can be split into Mod Configuration and Mod Downloader projects.
Note:
DOWNLOAD 1 is The download of the Mod-Info-File which is kept up to date by modders somewhere in the wide web. We know where to download via the with Wildfire registered Mod-Info-File-URL.
DOWNLOAD 2 is The actual download of the mod's data, i.e. all images, files, other content, ...
Note 2: The Downloader Project must resolve dependencies and also download those.

The overview is logically split into a section for:

• Future Mod Distribution Info for Modders.
• Mod Downloader Outline.
• Local Mod Configurator (all credit to leper, sanderd17 & Itms <-- pls don't be angry if I forgot anyone, just tell me).

---

Information-Collection
Important Info spread far:

• Mod-Info-File & Where to store it (either at Mod-location-side or 0AD-side)
• about Mod-Types & Categories

Locations: (Content, Functionality, Maps, GUI)

-> GameDataPaths

Map-Locations:

• "..put your [preview-]image into art/textures/ui/session/icons/mappreview/ and refer to it in the "Preview" setting in the map XML file."(fcxSania, thx FMZSchotter).
• put map into folder xy
• ...

Functionality-Locations:

• put script <here>
• ...

Content-Locations:

• put 3D model COLLADA entity <here>
• check if actor exists, sort out json file, put Actor <here>.
• ...

---

Mockups:

De- activation functionality in the middle part missing.
The right is for details as usual.
No idea if we need a modder ranking too. I would say no. Only the mod ranking.
The free top-middle region I thought for details at first because of long URLs not fitting in the right box.

The text field above the mod preview image is barely visible.

Hope you like it. Didn't like it. Files are on Github. <-- Have I posted it? omg. Github 0AD Mod branch.

Edited June 22, 2014 by Hephaestion

[Josh]

If you'd like a mod manager implemented, some design mockups and functionality guidelines would be a good start.

[Radagast.]

Definitely a good idea. I will edit the post once I have some available. First I have to hammer out more details about how it works, then I can provide the GUI. The other way round were contradictory, imagine you create the GUI first and realise later, good gracious! this is field is technically impossible or this option can't be user-defined. And such. I'm sure you didn't mean it that way anyway.

Functionality has been roughly outlined by Sander and my humble self in the Council of Modders thread starting from here (a bit spread and at first not very precise).

I think we could follow this guideline of feneur:

A mod manager should definitely be included in the main game, but that does of course not mean that you can't implement it. Please do discuss it a bit in the forums first though so it can be done well/work even for non-CoM mods.

We will see how it turns out. Haven't you been involved in the Lobby? There are some overlapping features like:

• world wide web access,
• data download.

Did you at the end already plan to work on it? We don't want to discourage you. It'll be highly appreciated as our list of features we dream of is long. Nevertheless I might come up with at least a intermediate solution.

[Josh]

I was heavily involved in the lobby, but I'm not exactly volunteering to make a mod manager (my hands are quite full at the moment). As far as functionality goes, I would suggest starting with a simple offline in-game mod enabler/disabler before going into more complex use cases (online functionality is a lot more complex than you may think).

Again though, starting with a clear design is a big part of making great software.

[Radagast.]

Hmm. You are right. This should be a separate functionality. De-/activating would come in handy if it were possible at a later point.

Originally I thought of showing up a selection screen with preview image and some rudimentary data like URL, type, mod pack belonging to ... this UI should be shown before the actual mod dirs are loaded. That's what I have to figure out how this is possible.

btw. Nice observer mode, just making possible a whole bunch of new possibilities, e.g. assigned two AIs to one player/tribe/civ however we call it. So you helped out my Hybrid AI efforts ...

Thanks for all this awesomeness in SVN, this 0A.D. is really even much more advanced than I ever had imagined.

It's really not far behind other famous strategy simulations, in some aspects even superior.

@mylist:

Will have to create a wall-demolition animation soon and try if I can prop it to damaged wall just on projectile impact (just like in stronghold).

Edited March 23, 2014 by Hephaestion

[sanderd17]

If someone is planning to work on this, you can take a look at ps/gameSetup/GameSetup.cpp. That's the point where the VFS (virtual file system) gets initalised and where the different mod directories get added.

Maybe this method should be changed to only load the public mod, make sure the public mod loads the mod-selection screen, and after that mod selection screen, load the other mod directories to the VFS. This should work together with some settings that remember your previous mod choices, and make it possible to start the game without showing the mod manager.

[niektb]

I did a throw to the design mockup:

It is taken from the Empire Earth tech tree modifier for scenarios

As you can see there are two columns. One with disabled mods and one with enabled mods. Using the buttons in the middle you can change priority and enable/disable it. Clicking the "+" gives additional info on requirements and stuff. Green means that the requirements for the mod are met, red means it doesn't.

It only needs a starting button, but I think it's a handy approach.

[Radagast.]

thanks for the mockup.

Hm.. I think I'd prefer to have to listen on click/touch event only. So it will be a list with many rows, including preview image. If you click it, you activate it.

If you click on a collection, the whole mod-collection(mod-pack, e.g. aristeia is activated and downloaded if it not exists yet in the mods folder).

Still this is a bit tricky and I'm struggling to orient myself in the engine. So it will need some time to get accustomed to it. (especially as my debugger is still non-functional and my symbols jump me to functions with crazy signatures.)

But don't worry. With our feature list this will be a long-time project anyway. And winter, our big friend, is still up to come.

Edited April 10, 2014 by Hephaestion

[thamlett]

i'm willing to help with some of the coding (out of my limited knowledge:))

also, if we need a separate server (other than git) i have one

[Radagast.]

Master thamlett is back! Hurray. Thought, you had been one of our officers lost in the last battles.

A Mercurial repository would be awesome for the artwork. Though this might be HUGE amount of data. 20 GB?

If we restrict our art collection on the CC0 textures only, then we can keep it in the git at first. Later if it gets more and more unmanagable, then we might choose your server if you don't mind. What else could we use it for? .. hhmm. I think we could host a Design Document tool there. I just have to first develop it. (if I do, perhaps a webdev joins us some time and might pick it up)

* hmm.. Server ... promotion? Lion might want to use it. Surely we will come up with tons of useful uses for it. It's getting serious.

The coding is quite strange, we need to find which functions we can reuse for

• Renaming files (mod de-/activation , prepending a point just like on *nix systems hides the files - in this case from 0AD => mod deactivated.).
• Sander said, we could start at Game.cpp if want to put the mod manager as a preprocessor layer somewhere before the main mods/public/ mod is loaded.
• Otherwise we have to find a better suited place. (this is a hard question with our limited knowledge of the engine). We need know all the interconnections to decide where to put it. Time will be our friend here.
• The UI we could start already. Will need some JavaScript. We define JSON-files for each mod as the engine has to know where the mods in the internet are located (=> we need to provide a URL, sander and me settled on a preliminary draft in the link posted as a answer to Josh).

[Radagast.]

server going crazy

Edited April 13, 2014 by Hephaestion

[thamlett]

well lucky me, i have nice high bandwidth and no data cap

[Radagast.]

Going crazy too now. Spending years trying to post long ones ...

(that's good, at least you will not fall victim to the re-edit prevention. If you edit your post and use the BBC editor, then there are plenty bugs around: breaking quotes, random insertion of list entries, re-submit time-out, deletion of lines .. format destroyed ... sometimes I forget to copy the text into clipboard or somewhere else. Then all is lost. => redo formatting .. wasteful)

Trying to decide if XML or JSON shall be our data structure. Feel symphaty towards XML as there are unique paths (xpath) (allowing interesting solutions, one of which I will hopefully conrtibute in a few weeks). XML is handled by Xeromyces in pyrogenesis C++. As we have to deal with heaps of files we have to deal with another C++ module anyway (VirtualFileSystem).

We could use JavaScript to parse JSON but it has to be delegated to the Engine anyway. A lot of data passing forth and back then. So I think the JSON has disadvantages (though it's cool too). JavaScript had to do the renaming of files via the SpiderMonkey anyways (better rename than have to alter the mod-config-file's content).

Any inputs?

Our base is Sander's structure. I recall here:

Instead of Bash, we're more used to JSON. <-- preprocessor is off the table.

Something like

{  modName: "Millenium AD", // name of the mod that will appear in the GUI  engine: ">0.0.15" // Minimum version Alpha 15 required  modVersion: "0.0.1",     // version of the mod  site-uri: "http://moddb.com/myMod" // link to mod information page  source-uri: "http://my-server.be/milleniumAdMod-0-0-1.zip" // Download of the zipped mod  depends: ["someNiceGuiMod >= 2.0"] // list of mods this depends on, with optional version information (using >, =, etc to denote minimum and maximum versions)  sha1sum: "..." //<--shall the user generate this manually and enter it here? I think that's no good idea.  // other keys: "provides", "conflicts", ...}

For users, it's still safe if people can only modify their own mod file, and by using the sha1sum. We can also make sure that the mods are loaded in the order required by the dependencies (and alphabetically otherwise).

The scheme is definitely what we need. Though the shasum I don't like. Only modders can mod their files and to check the downloaded file's correctness is impossible via the sha12sum because which modder will update the SHA sum in the 0AD sources everytime a change is made?

The base file name should probably be the same as the mod directory it installs to, to avoid mods overwriting each other, and the file should probably be present in the mod root directory (so we can still check dependencies, even when offline)

If file already exists, I would append a number simply. That's easy to remove for nicer displaying later.

Recalling more posts for us not having to over and over look for the relevant parts spread out in our large and diverse Council topic(s):

To be clear, we need the above Scheme on the 0AD side!

For the online side I would prefer not having an extra file, instead deriving the mod names directly from the folders/filenames. Categories like standalone: functionality: theme: could be done by prepending to the mod name. Hiding a mod can be done via starting the folder name via . or #.

The icon can be named like:

<modname>.png|ico|jpg|svg|gif

The overriding and put into folder concept is good.

To solve the dependency issue we need a resolver for that. Then of course we needed a configuration file anyway for reading in dependency. We could call this

mod.xml

But this makes it messy to resolve doubles as the file first had to be read while the folders itself already prevents doubles (as there can't be equal named folders). But perhaps we should allow for identically named mods?

The question is whether to set up a mod directory/folder somewhere highlevel within 0AD folder structure. e.g. data/mod/.

Or if it's better to simply add mods via putting them into the current folders. This of course would be a requirement for AI mods and also it would simplify things a lot as the program has not to read in all those mods from the mod folder but it's important to mark those models/actor files that are mods like infantry/<mod_pack>:citizen_soldier_...xml so that we can keep track ofwhich mod belongs to which mod pack if we'd choose to remove one on the fly.

Oh wait, the last one is superfluous luckily as we can solve it all with a preprocessor:

• instead of launching 0AD directly we introduce a
• shell/batch script that is aware of the location of the 0A.D. files and has write permission

  that performs:

• show mod list of all available mods (grouped in mod pack categories, e.g. Aristeia, 1000 A.D.), (resolve dependencies in real time after checking a mod using JavaScript)
• download of mods that have been selected into /tmp folder,
• extract it into the correct folder within the 0 A.D. file structure.
• mods are loaded during startup.

I could and just decided to write such a script for us (at least for linux) .. <- tight integration is desired. => no processor.

Mods can be spread out over the internet. Must be directly downloadable.

Depending on the mod pack (e.g. 1000 A.D., 1000 B.C.) another variant of e.g. our hybrid AI might well be chosen and downloaded by the script.

A mod pack uses the same config file structure and almost exclusively consists of dependencies. => Organise your mods like you want. Even without the 0AD file structure. => The mod manager has to sort it correctly into the individual folders. (We will not deal with archived files at first as this hampers moddability and reuse as you can't reuse a file in a zip-Archive in another file - the downloads could be extracted automatically though. Let's see. Is not important for now.)

Reusing scripts/functionality is possible by strictly distinguishing

• functionality (e.g. a campaigning engine) and
• the relevant mod content, e.g. viking campaigns, roman campaigns, ...

e.g. We want exclusively Roman era mods in the Roman Republic repo, i.e. additional custom roman units (like standardbearers), ... Roman campaigning content files, ... still we can use the same Campaigning functionality by marking it as a dependency in the campaign-content-mod that goes with the mod-pack (e.g. 1000AD).

The importing via xml, e.g.

<import>    <mod-pack>Roman_Republic</mod-pack>    <!-- OR -->    <mod>1000AD/path/to/mod_folder_or_file/<mod></import>

allows us to keep it all separate - no redundancy. The script should take care of downloading the imports and resolving additional dependencies!

To have the url-encoded source- or download-URL as the filename allows ultra-quick sorting. Could also put the mod name in there. What I don't want is to have a non-saying filename. Filenames have to be unique anyways, so you have to care for how to name that file anyway.

The URL can be the filename. It is unique and the might change seldomly. Though me personally I prefer a descriptive mod name (not 1 not mymod not changed_number, ...).

We can have tags/description though. The filename can easily be parsed into a display ready name, e.g. "_" to " " conversoin or CamelCase analysis. It doens't matter if the people have to care for defining a well-thought name inside a JSON or XML tag. Or if they need to think of a well-thought filename. I prefer the latter.

Dependencies:

At first I thought we could resolve dependencies via SHA1|2 but as the file changes, it might still be compatible, so only equal is possible and not >= <version>. => so we need a version number. Might be extracted from the filename (Mod generic name).

How about changing the dependency check entirely to:

• Analyzing the type of the mod (e.g. sword, bow, ...).
• Analyzing the generic or specific name (e.g. Pilus Spear).

So if nothing were to be found we still could fall back to using any spear. Would be quite handy and the more natural way too. If You want a pizza then you also don't ask 'hey the pizza is not looking like I showed you in the picture, it's darker!'. The answer would be: 'Well, the function is still the same. No pizza is identical.'

If we adopt this for resolving mod dependencies we get a rigid system. e.g. in the case of a map:

You want the Acropolis Map of 0 AD. well it could not be found, the Mod is broken or what ever. As the mod manager analyses the function/target of the map, i.e. Greek and ancient, it can simply choose the default map as it's Greek and Ancient and meets the requirements (dependency).

So each mod could fulfill a function/target and this has to be specified anyway in some configuration file. We could simply reuse this configuration file as the <mod>.xml. If so then we have to add optional xml-tags:

• download-URL (simply url does it.)
• ...

Still the organisation as is currently in 0AD can stay unchanged. i.e. If launched with

pyrogenesis -mod=<mod-pack>

. Then all the dependencies are being resolved. Finally the relevant mod packs (where the needed mods are located) will be activated (this time not all overwriting) or even better the mods are handpicked from the mod-pack's subdirectories without having to care which files overwrite which.

The order the dependencies are listed in the mod's configuration file determines the order in which mod-files are loaded, overwriting existing ones.

[Radagast.]

To clarify. Research showed, we have to decide upon where we store the special data, either:

• In a information file on the 0AD side. If modders change things, like category, they change it on the 0AD side. (and changes to take effect takes until the next 0AD release, not so nice, will produce Modders setting up their own system)
• The information file goes to the mod files. On the 0AD side we still need a e.g. from a database generated text-file containing all mods, in form of a URL. If we instead want to have a file for each mod, then I would opt for making it a file containing:
  • Mod name as filename.
  • source-URL inside,
  • metadata inside. (limited information and generated in contrast to above solution 1)

In the latter case we have 2 files, one that points to the source-URL and maybe base64 encoded preview image inline? Also some metadata that changes seldomly, like type, function, ...

We here have all variables that change seldom with 0AD. (so modders don't have to come by often to change it, e.g. Version number whenever they change the mod). This extra data like dependencies then goes into a file that comes with the mod.

The first solution saves us the file with the mod. All data is on the 0AD side, so if something changes, e.g. version/dependency, features, then the modders need to edit the data (either URL&type-list or the individual files) in the 0AD repositories.

For the case the URL changes, of course as we can't guess it, the modders have to re-register their mod or change the link somehow.

Personally I don't prefer any solution over the other. Just easy to get confused and thought we should clarify.

Concerning the Mod name as the filename:

I think as modder already have to be aware how the 0AD structure works (folders, .. which has to contain what to make it work, etc.) it really might not be a problem for modders to stick to filename pattern. Just saves us reading one xml-tag or json-entry and limit the mess (e.g. 100 files named mod1.zip).

---

Sidenote: Does anyone know if we have functionality for file-download already. If not I have a solutoin. If yes, please help me out and show me where and how you recommend to use it.

[Radagast.]

I'm stuck with random crashes: list_name ... okay. Now I have it. Just forgot how I fixed it. omg.

[Radagast.]

Updated & tidied up the first post: -> To the first post.

Several discussions with MuteLovestone, Lion, other councilors & on IRC have brought us closer to a decision on where to store what. See here for the discussion (also linked in the first post).

MuteLovestone wants to see how much information for a Mod-Info-File can be automatically generated.

Edited April 26, 2014 by Hephaestion

[Itms]

Hello everyone, I have been thinking recently about such a mod manager system. If I can give you a hand, don't hesitate to tell me!

I have a few ideas about how to realize in in practice, but in general I have problems concerning usability and clarity for the user, so a whole team of modders with specific needs would be perfect ^^

For the beginning I think we should implement the "manager" as a part of the public mod, like we do when we launch Atlas from the main menu. When it is functional maybe we will be able to create a meta-mod, but this is another decision because 0 A.D. is after all the main game here, and all (y)our work of modders depends on it

Edited May 7, 2014 by Itms

[Radagast.]

Itms, that's good news. I already liked your patch which free the way for the Mod Manager.

It has not to be separated. I'd rather like using libcurl and building it directly into 0AD. Actually I already created the GUI ingame draft and added a 'Modifications & Extensions' button to the main menu.

But the GUI is quite volatile currently.

With libcurl we had cross-platform download and upload capability, though I think we already have that - yet I have not yet fully grasped NetFileTransferer. Probably we even have NetIO.

But you are right, we need to adapt the mod-folder loading. It should be repeatable. Currently the mod folders can only be joined once. But if we want to have it ingame (and not as preprocessor) then it would be best to merge/overload mods on-the-fly -- whenever the user chooses SAVE within the Mod Manger.

Thus:

• Open /Change GUI page to Mod Manager,
• There a list of all Mods is loaded by downloading the Mod-Info-File list(s) (multiple if we want to give the user the chance to decide whether he wants to download Mod-Info-Files of official, community or unchecked mods.).
• The user takes a selection. We listen to it via the onTick event.
• We check the dependencies (which are noted in the Mod-Info-File we just downloaded on entering the Mod Manager GUI page or trigger manually via separate onTick events) and activate/select those dependencies too.
• User clicks SAVE button.
• We now download each mod one by one now.
• The downloaded mods now have to be merged/overriding the already existing/earlier downloaded mods (e.g. mods/public/).

  <-- we need to somehow figure a loading order but probably we have to load the mods by alphabet as it is currently

So far my research.

Of course we could just like Sanderd discussed with us modders earlier allow the selection of mods only once per game start. Then we have to hook this "Manager" functionality not into a separate Main Menu GUI page but somehow inbetween the loading process.

Then we also should add a config option to deactivate the pop-up of the Which-Mods-Do-You-Wish-To-Select-Screen on every game start.

Your recent patch for splashscreen is what we could reuse here in anycase.

[Radagast.]

Now in the IRC leper & sanderd17 opted for the allow-mod-selection-only-once per game start solution.

And this will be before we load additonal mods. and probably after loading the public mod. As we need the GUI files.

If you want to change your active mods, just restart the game.

An alternative solution came in:

@leper> maybe have a mod that has a page that allows to select the mods when no mod is specified

21:36 <@leper> save the last chosen mod(s) to the user.cfg file, and load those on game start

21:36 <@leper> (maybe with an option to load the mod selection screen)

21:36 <@leper> (and add the mod selection screen to a menu in the public mod)

So that I don't forget this. leper really has come up with something ingenious, holy blue planet. I can hardly grasp it.

http://irclogs.wildfiregames.com/2014-04-28-QuakeNet-%230ad-dev.log

Edited April 28, 2014 by Hephaestion

[Radagast.]

First post updated with an Overview of the current state:

Split logically into:

• Future Mod Distribution Info for Modders.
• Mod Downloader Outline.
• Local Mod Configurator (all credit to leper, sanderd17 & Itms <-- pls don't be angry if I forgot anyone, just tell me).

=> It seems the Mod Manager project is splittable into Mod Configuration and Mod Downloader Edited April 29, 2014 by Hephaestion

[FueledByOCHD]

I will start workin on the program to make the mod info files (give me about 4-5 hours, (if that))

[Lion.Kanzen]

Don't forget load screenshots about your layout and your interface progress. ( for promotion reasons of course )

[FueledByOCHD]

probably should do that for my new program i just made, it make creating a update log alot easier

[Radagast.]

Here is a somewhat bad screenshot of the new approach (leper's invention).

leper has made progress while my GUI still sucks. Will work to make the testdata show up and to get the sorting and enabling and removing debugged. all the functionality exists but it seems not to work yet. :/

[thamlett]

looks good, rada

the buttons might be smaller though