Jump to content

Sharing analysis results


Recommended Posts

Hello,

This is some results that I got with the non-visual autostart :

The settings used :
    - All results are on the map 'Acropolis Bay (2)' (no modifications).
    - Only Petra players.
    - Only in conquest mode.
    - Only 4 civilizations used.
    - 200 games for each result (one row in the table beneath), with an AI seed from 0 to 199. (Some games didn't finished, 400 000 turns and still continuing.)

Players win percentages with the specific civilizations :

PctVictPlayersByCiv.png.c62cbd7f89d5f9c3dc1d917528050109.png

All the plots (made with R language) showing the evolution of the win percentage for the first player (ID=1) : Plots.zip

This shows the win percentage for a civilization against another one on the Acropolis Bay :

PctVictForCivs.png.bbaa3fb7250b3fe4993286d348540ef2.png
Note that this is only with Petra players, and if I'm not mistaken, Petra doesn't adapt its strategy in function of the civ its playing.

This shows the advantage of the starting position for a specific civilization on the Acropolis Bay :

PctVictForPositionByCiv.png.062e8387616e038d3fc14aacb5326c01.png

I wrote a Java program to run multiple games, save the summaries and analyze them. But it will be better if the the possibility to run multiple games and save the summaries in a file is integrated directly in 0A.D.

 

Possible improvements for the non-visual autostart:

Like mimo said,

- add an option to run multiple consecutive games with the same settings but different seeds.
- add an option to impose the same civilizations on all players.
- write the summary in a log file as a JSON object, to be easier to parse later. And when you run multiple games, write all the summaries in the same file, maybe in an array or something like that.

I thought of a JSON summary that will look like that :

{
    "aiseed":0,
    "map":"Acropolis Bay (2)",
    "victoryconditions":"conquest",
    "victoryduration":0,
    "totalTurns":15000,
    "players":[
        {
            "playerID":1,
            "ainame":"Petra",
            "aidiff":5,
            "state":"won",
            "team":-1,
            "statistics":{},
            "civ":"brit"
        }],
    "extra":[]
}

The "extra" attribute will be an array of object where we could add additional informations, and if possible from anywhere in the code, especially from the AI. For example, the AI I worked on, uses different strategies over a game, and it's important to know what strategies were used to analyze them later.

And maybe, if it's possible and useful, parallelize some games, to speed up the process. This will also help not to block all the games when there's one game taking too much turns, as long as all the games running are not endless.

  • Like 7
Link to comment
Share on other sites

Nice work!

Care to publish the additional tooling somewhere? Also I do hope you are interested in working on some of those improvements.

About games being stuck, it might be interesting to save the game at that point so they can be inspected manually (as the AIs might be in some state where neither can win, but none is defeated). Proper (as in fully deterministic) AI serialization would help with figuring out some of the internal details at that point, maybe we'll get there someday.

For the summary that looks quite a lot like what we do for replays already (and part of it would be the first line in commands.txt files). About the extra parameter I'd both appreciate having that per player and then kept in cmpStatisticsTracker and updates to that being done via simulation commands.

As for running multiple games in parallel I'd do that via some sort of launcher, so we don't have to worry about possible non-threadsafe code in tons of places. So the only issue we'd be left with would be file system access (and that should be read-only for the most part, except writing the statistics (and that should just use pid+timestamp), but that should be done to one file per game, so there is no huge issue there). Running a few games consecutively could however be done by the game relatively easily.

  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...