[Tool] Design Document Parser & Civilisation Editor for XML/JSON

[Radagast.]

I'm trying to set up a database for all civilisations (at least for mods as there chaos is great king).

[This post will be updated.]

Where to find:

You see an old rusty gate with foreign inscripts. Pass through it. (the UI is not really functional yet, as no data exists)

Terminology:

- Leaf node: Mostly text. The final value.

----Civilisation----civilisation_file.xml:<root_element>    <element attribute="leaf node">        leaf node.1    </element>    ...<root_element>

Functionality & Status:

- Generate a civilisation or mod structure from an existing template, e.g. an existing civilisation or from a mod folder recursively (such that it has not to be created from scratch).

- There is no fixed export filetype. Both JSON and XML can be output (to put into mods/<mod>/ folder). untested

- The export target fileformat can be set on a per file basis. And changed even for already existing civilisation data.

- Add new civilisation.

- Rename civilisation,

- Merge civilisations.

- Add new File to civilization.

- Rename file within all civilisations at the same time.

- Move elements from one file to another.

- Add new leaf element. (via xpath)

- Insert/modify leaf node values. (good for researchers that don't have git or repository access)

- Version control. not yet

- Sorting (after Civ, File, XPath, or Leaf Value),

- Grouping (after Civ, File, XPath, or Leaf Value),

- Filtering (dynamic),

- Option to apply changes to all occurrences in the whole database. <-- That's powerful, but also very dangerous if it goes wrong.

- Option to modify equal values through out the currently filtered data rows.<-- That's quite useful as this way you can 1) filter the rows you wish to change. e.g. all occurrences of mesh_and_animation.v1.dae to mesh_and_animation.v2.dae within a certain civilisation..

- Updating the database civilisation structure by reading a template folder recursively when data already exists, creates new leaf elements automatically and shows in the civs that new elemets now have to/can be added/ are still empty and need to be filled. buggy

Example: A new required element has been added, e.g. timetag/timerange. Now all civilisation data that not has this element yet, will have this added automatically (with dummy data, that can be changed thereafter).

Targets:

The target is researchers. Hopefully this later can be combined with a mercurial setup. I just have to add a button to allow researchers to commit their data.

Edited May 12, 2014 by Hephaestion

[FueledByOCHD]

Well maybe i can make a version of this in c# then you can end up converting over to web? just a thought,

Awsomely,

Mute Lovestone

[Radagast.]

Well maybe i can make a version of this in c# then you can end up converting over to web? just a thought,

As I use SQLite for the database (1 simple file) exchanging data should be really easy.

Synchronising is what I'm concerned of.

Local version:

Benefits

- A local version had the benefit of being able to directly change in the mod filesystem.

- And if you make a change in the XML via commandline or text editor, then this could be synchronised easily.

Diadvantages:

- Yet people have to download your program + .NET/MONO runtime.

- And they have to commit the result somehow in a repository. For that it's necessary to have a versioning program. Also conflicts have to be resolved with delay. Probably a dead end for a researcher.

Alternative: Local web-program execution:

With Backend: Set up a local Apache e.g. XAMPP for PHP, Tomcat for JSP and run our web-program locally in your browser (localhost).

Live civ editor for 0AD

Client only: If we converted the PHP to JavaScript and ran it directly in 0AD - a live civ editor for 0AD were born. It could then be executed locally without even localhost (and have file I/O access via SpiderMonkey-->Pyrogenesis).

Reasoning for why pure JavaScript only is sensible within 0AD:

A purely client browser program, i.e. without backend, had a problem: it's impossible then to read or change a file in the 0ad data/mods/ folder.

For the database there were sqlite db engines or even pure JSON storage. <-- so the database is not the problem.

But the benefits of local are all but no internet-connection lost as files can't be changed. While the disadvantages remain.

=> A local JavaScript only version only inside 0AD.

(our tool needs to be able to have I/O for the scheme/folder structue import. )

Result:

It might be useful to have a lightweight from 0ad separated program at hand for research. On the other hand live-edit in 0AD for researchers would be epic.

For live-edit from within 0AD the only disadvantage is that 0AD has to be run - and might be memory hungry.

---

Usage:

Example -42

Most useful might be the up-to-date keeping of the xml/json format (change a xpath for one xml value in one file and the same happens consistently for all files with the same xpath therein.).

Example -1

Or assume 0AD has changed the element scheme. Oh no!! It's all invalidated now! What shall we do? Update a million civilisations' data structure manually? As we don't want our beards to grow too long while editing, the least we could do is changing one civ and the other civs follow automatically.

Example 1

Or if you rename file ai/data.json , then all existing ai's data.json will be renamed (identical values are linked by default). We have a copy option to make single user copy. Then delete the old data row/value and we have it: a new file data-bloc which we can rename individually, e.g. ai/data.json -> ai/data.json.bak .

Example 2

Also if you group by xpath you see which content other civs have close by. Nice if you want to know what value similar elements have.

Example 3

Once we have a timeframe tag ingame in our schemes, then we can achieve a nice folded view sorted by year (of all our data):

Receipe:

• group by timerange /begin or timetag,
• sort by the same column ^

.

And the best is, it's not really complicated to achieve. Otherwise it wouldn't be worth it (as we can achieve the same with UNIX commands). As said before, it's not for the Linux expert, it's for the researchers.

Don't forget me:

We have to add a check that checks if the value has been changed by someone else in the meantime. These rows should be highlighted and the old value be updated - or a triple diff shown (but that are really things we add bit by bit with time's river flowing uphill).

Edited April 27, 2014 by Hephaestion

[FueledByOCHD]

I would say yes due to fact that if we add to much to the pyro. Sooner or later massive lag will follow

[Radagast.]

Again it took longer than expected, but I'm working on it. Will continue the whole night.

Now we also have a filter.

Import is still untested. Export non-existent. But slowly we get around the

sorting,

grouping,

ordering,

adding Files

and copying elements

as well as adding new civilisations.

[Radagast.]

I would say yes due to fact that if we add to much to the pyro. Sooner or later massive lag will follow

This lag could be negligible. Still we could add it to Atlas. Though this is no longer live edit to see ingame how the Descriptions read, ...

I/O while ingame could also be a problem . But if it is a 1-Player test map only, then it could work without lag.

*Update*

now we get closer. still some crazy javascript bugs to fix though.

Edited April 28, 2014 by Hephaestion

[niektb]

When trying I noted that the console gave an error while he tried to load the jquery tablesorter:

Failed to load resource: the server responded with a status of 403 (Forbidden)

And so I couldn't test anything further as it break at creating a new civ.

[Radagast.]

Yeah that's puzzling me for days now.

That's coding. It's working here. It's not working there. I have checked all preconditions. Have debugged it and all scripts are loaded.

Also the left side of the table should be green. The reason for this could be a broken image or conflicting CSS styles. It's easy to fix but it's a bad sign because it should work out of the box as it's not the first time I use it.

Edit: The tablesorter just doesn't work because there is no data, or only inhomogen data rows. I have to work one more night on it to get it all running.

When trying I noted that the console gave an error while he tried to load the jquery tablesorter:

Failed to load resource: the server responded with a status of 403 (Forbidden)

And so I couldn't test anything further as it break at creating a new civ.

Ah dude, indeed that could be the reason. I noticed the permissions are screwed before. But never thought it was an internal problem too. Edited April 28, 2014 by Hephaestion

[Radagast.]

Thanks for reporting. That's the problem indeed, must have happened while uploading the file.

[Radagast.]

thx man thx thx .. I had already fixed this error for the pager (page-switching) ... but me fool didn't realise it could be equal. How much time have I wasted on that. I'm so sad. thx again.

[Radagast.]

Now I have to import the blanko or filled template civilsation. I need to extract all files that belong to civilisation somehow.

we could also use the complete mod folder but for testing and for research that's not feasible. Because some values need programmers and artists to specify them ... the researchers could of course simply skip those. I dont know what would be best though.

Also note that there is a JavaScript problem if you want to add a new data row (via the yellow button).

Edited April 29, 2014 by Hephaestion

[niektb]

Thanks for reporting. That's the problem indeed, must have happened while uploading the file.

You're welcome.

I noticed a couple of other errors too:

Uncaught ReferenceError: screenshotPreview is not defined (at line 404) => directly at startup.

Warning: event.returnValue is deprecated. Please use the standard event.preventDefault() instead. (but that is in the jquery script)

Uncaught NotFoundError: Failed to execute 'insertBefore' on 'Node': The node before which the new node is to be inserted is not a child of this node. (at line 268) => when clicking the yellow '+'.

[Radagast.]

Yes that is what I meant with the yellow +.

thx.

The good news is I just succeeded in fixing it. You can test it, it should copy the data row with content now.

Your hint that the file couldn't be read has given us a boost. I have to activate the console's NetLog from now on. Thx for the reports. We're moving forward. Next I have to fix the civilisation update. It's already three quarters done but it's complicated.

[Radagast.]

uhhh. there is a bug because of caching in tablesorter:

Whenever we add data rows and then change the sorting/grouping (grouping via doubleclick) then we lose all the new data rows.

Actually rather a frustration generator for researchers than a help!! :/ Have to fix it in the jquery ... will do.

[FueledByOCHD]

What git is this in?

[agentx]

> A purely client browser program, i.e. without backend, had a problem: it's impossible then to read or change a file in the 0ad data/mods/ folder.

You can. Latest browsers provide a drag'n'drop api and a file open api, and to save you can provide the file as 'download' and the user chooses where to save, which is even better than directly overwriting files. There are good plugins available for both, shouldn't be an issue.

[Radagast.]

Oh, you're too quick for me. Not yet committed. And I should be more careful with committing unfinished works.

Btw. is it useful? It's a real mess. (but I'm working on improving it, then I'll commit it, is that okay?) If I shared it now then like with Mod Manager mock-up people will wonder what I was doing.

@agentx:

Oh, didn't know. Must reconsider then.

Perhaps we should even go without I/O and just pop-up a text field with the generated XML/JSON content?

Edited April 29, 2014 by Hephaestion

[FueledByOCHD]

Sorry I have no net ATM should today

[Radagast.]

You're really funny man. It's nice to have some status messages. Me personally I try to delete mine once they've been read, but well .. your's are funnier.

Several new features for the Civilisation Editor:

• Option to apply changes on all occurrences in the whole database.

  <-- That's powerful, but also very dangerous if it goes wrong.

• Option to modify equal values through out the currently filtered data rows.<-- That's quite useful as this way you can 1) filter the rows you wish to change. e.g. all occurrences of mesh_and_animation.v1.dae to mesh_and_animation.v2.dae within a certain civilisation..

It's almost finished, I hope for a breakthrough tonight. <-- today did not work, new attempt tomorrrow Edited April 30, 2014 by Hephaestion

[Radagast.]

The data rows now are kept up to date. So that we have neither losses nor a messed up sorting when we add new data rows.

Btw. Note that usually when we add a new data row (xml element + value) then we do this for all civilisations. So that we don't have to do this for each separately.

This way we automatically know where values are missing. Best we highlight empty rows with red backround. Have to investigate.

[niektb]

When trying I got the following errors:

Uncaught ReferenceError: screenshotPreview is not defined (at line 434)

Uncaught TypeError: undefined is not a function (at line 288)

[Radagast.]

Thx, comrad. The first one is a feature. (which we will not have too early, but so that I not forget about it we have this error, a bit crazy I know, perhaps I better remove it)

The second one is new to me.

Also a submit button is missing. Without this button noone can save the changes.

Have to test the loading/saving now. Will load our eastern friends mod.

[Lion.Kanzen]

A Add button, factiontionality because the number of units in a sci-if or fantasy based in warcraft it is not same from our Historical based.

Add new CS( citizen soldier) Add new su( support unit) Add new heroe, add new ship, new creature ( fantasy ) Add new special unit

For more flexibility

[Radagast.]

Absolutely that's what this tool is about. And if we have to add the Nomadic Pack Element to XML files of a certain type (be it as optional or not) then we also use the already existing Yellow Add button at the left side.

Doesn't it show up for you? <-- essentially when you sav after adding a data row, then the other civilisations are being synchronised.

Still have jquery /tablesorter problems. I had never expected so much time spent into those issues as I have it running perfectly in another project of mine. Hopefully this night will turn out better. <-- that's history. Yet jquery caching is giving terrible problems.

Edited April 30, 2014 by Hephaestion

[Radagast.]

The Modders have settled on harmonising MuteLovestone's & my humble efforts. It's best to clarify:

• Mute's Modding Toolkit is for Offline modding.
• Mine is for organising and upkeeping of the modding structure. To keep mods in synch (with e.g. optional element tags/attributes) and to give The Council/0AD team a nice overview of which values yet have to be filled.

Hence we had to close doors and a Password is now required as we now pickup development with Scion Development's Rise of The East which Niek &amp; Rob Kimball discussed for some time already. Also it's a highly vulnerable system. We can't give access to the whole planet as bots might change our values and templates randomly.

We're still just an umbrella for modders. So if this tool might prove useful (it's not yet clear) then feel free to request us to maintain a mod for you or join our efforts by showing you really want to mod. Don't worry, there are plenty tutorials spread throughout the forum. Starting modding is easier than you might expect, changing values basically. (proper research, adding features &amp; artwork is another topic)

Edited May 1, 2014 by Hephaestion