wraitii

I never realized how strong the women were in the new meshes haha. Amazing work!

Looking at range queries. It's about 10ms to execute active queries in the late game, because we have 500 active queries. In general, parabolic/global/regular queries are about as much time, though their slowness varies. For general queries, we might gain some ms by actually checking for max distance before we check for component existence, since that seems to be slightly more efficient. The good news is that the subdivision Move/add/remove are mostly irrelevant in terms of computation time compared to the other stuff. We might be able to speed up queries by just checking against units that moved this turn. This needs to be tested and probably needs to be optimized spatially a bit but could be interesting.

Fairly more comprehensive profiling. The three attached graphs are a very smoothed, less smoothed, and proportional plot of the same game. As you can see, we have the occasional spike in LongPath, probably when we move many units not in formation. ShortPath also occasionally spikes when units move around, and particularly when they combat (which can be detected because Timer, which is the JS component mostly used by unitAI, spikes too). But as a general rule, the longest call is the update of Timer.Js, which afaik is mostly the gathering, repairing, and combat timer. (NB: profiling are exclusive, Timer is part of Components in the above graphs).

Some more detailed profiling, based on a 2V2 provided by Elexis (which is basically using rev ~ Alpha 21). This is an MP game so turn rates are 500ms. All analysis done using a custom profiling script that outputs per-frame data and R. First we can look at the whole thing, smoothed. "Components" is Update Components, the rest are various pathfinder related things. I don't profile long-range at all here. Then unsmoothed: as we can see there are several very high spikes. Indeed R tells over over a third of turns are over 50ms, over 80% are above 20ms (which is 50 FPS if we ran a turn every frame - obviously we don't). The SP turns are about 200ms, also 50 ms per turns means already quite a slowdown. We can do some fancier things, such as find the average breakdown for buckets ( <20ms, 20-50ms…): As you can see, the short-range pathfinder is hardly the number 1 cause of lag. The number 1 cause of lag is in fact updating components. I'll profile some different things with this replay and upload a new post for more info.

Been looking at the Combat attacking timer using Combat Demo Huge. The component update itself is around 100ms/turn at peak, so the pathfinder is not the only issue there. Specifically on the combat timer, finding new targets is by far the slowest, then it's a combination of the rest. Could be optimized by removing redundant checks, mostly, and improving QUeryInterface I guess. A particular source of slowness is TargetKilled, because of the two calls to the statistics (I think the query interface through player ID are particularly slow). Interestingly, the slow part of finding new targets is the JS AttackTargetByPreference more than the cpp, which is reasonably quick in comparison.

Are you working on those to upgrade our 'rise from the ground' stuff? And if yes, could you also add the scaffolds as actors?

Re 3D-mesh mountains: I do agree that something like this would be good for the game, it would allow us to give more details to the terrain where it is currently quite annoying to do so. Likewise for cliffs. I'm thinking maybe a better solution however would be some sort of 3D-decal, where we have a model of "pikes and rocks" that just overlays the terrain dynamically, so you can have details for various types of mountains instead of making them similar. More like a special terrain tile than a mesh. Though for cliffs you should be able to get away with the "follow height" actors.

I don't really think this is survival of the fittest, though. That's a fundamentally PVE game, I'm definitely thinking of some added element to PvP or PvAI.

Ran some more profiling, it seems PerformGather is really slow when the Fogging activates (which isn't really surprising), generally because copying information is slow (presumably because querying all those interfaces is a little slow). Not entirely sure how we could improve the data copy though. For the rest, the Gathering timer could be improved by removing a few checks in the beginning (in particular CanGather check seems useless since PerformGather does it again). Sending stuff to stats and the AIProxy takes around 0.2ms per turn, which isn't trivial but doesn't seem worth optimizing. Looking at other Timer calls, the important UnitAI ones are Combat.Attacking and Repair.Repairing, so those are the targets.

Looking more specifically at Gather, it seems the slower parts are the following: -The initial call to "CheckTargetRange and CanGather". Can probably be slightly optimized. About 1ms at turn 1500. Probably slow because called so often and may call CPP -The case where we need to find a nearby resource or dropsite (haven't looked in detail but most likely the first). -The call to return to the nearest dropsite when we're full (about 2ms) -The actual call to cmpResourceGatherer.PerformGather. This is the big one, 5ms on average at turn 1500. This is annoying because this last call is slow mostly because of: -The PostMessage "ResourceCarryingChanged" -The PostMessage in ResourceSupply "ResourceSUpplyChanged" -The call to Fogging We should consider directly calling AIProxy for those, perhaps that would make it faster. Another thing we could do is wait for turn end to send the resource supply message, since often a resource is worked on by many entities and we might avoid sending repetitive messages that way (unsure). Another thing is that in MP in particular the timer may get called several times per frame, and we are not "clever", we call everything twice, which is wasting resources. We might win some time by calling the timer less often. Not sure exactly how often that happens with resources though, but if it happens it's late-game where lag is already bad, compounding the issue.

Opening this thread to post some profiling results for my own sake and if anyone cares. It's not going to be very formatted Results after the 2000 turns, some at 3000 (i.e. after the 15th minute, we can reasonably assume players are skirmishing and have a solid amount of pop by then) I was looking at which components and messages take the most time. Using an MP replay of a 2v2 for these. In terms of component Guard is an obvious bottleneck, taking often upwards of 5ms for no good reason. It needs to be rewritten. UnitMotion is fairly slow (7ms). UnitAI on its own is somewhat slow. And when they show up, range manager and mirage can be slow. The slowest is by far Timer.js though. Looking at that one specifically, it's because of unitAI. Though Damage calls can be slow too. For unitAI, it's mostly because everything is too slow. The Gather timer is easily 7ms per turn, Combat when it shows up is terrible, Repair is slow too. Everything needs to be sped up.

It's more like the barbarians in Civilization. You're playing normally, but if you have uncontrolled ground in the FOW, you can get barbarians that will harass you. In 0 A.D., we could occasionally spawn raiding parties of 3 or 4 soldiers. Enough to disturb your farms a little if you're not careful, or your miners outside the base, but not a huge deal.

I don't believe I've ever read this idea as I've put it below: How about we introduced the concept of "Barbarians" or rogue units in 0 A.D. ? If there are parts of the map that are not in any player's territory and under FOW, they can span small bands of gaia units hell-bent and going to your farms and raiding you. Nothing impossible to deal with, but would keep you on your toes. Possibly as a new game mode?

Completely agreed. I think we should lower their size (pathfinder and real), and make them use melee attacks against other ships.

This looks like a visualisation of the svn logs. Rather interesting.

Hey wow, thanks for the notice, points 2, 3 and 4 should be fixed now. Point 1 is actually much more difficult to do than you'd expect, so I let it out for now. Very interested to see what you can do with this, personally I think we should use this feature quite a bit. It sounds like it could lead to pretty good stuff.

What I mean is that we have "formations". So either you put those 150 units in a formation then moved that formation, or you just selected 150 units and told them to move. Either way this is probably the short range pathfinder and there's not much that can be done right now sadly :/

Were you playing SVN or Alpha 20? edit: nevermind, Alpha 20 it is. This is going to be more difficult to debug, but at a glance to the commands file the odd thing is the walk command is repeated 3 times. I'm guessing the slow framerate is because of the short-range pathfinder for 150 units running into each other. I'm assuming you did not use formations for this?

That sounds catastrophically worse than what you should experience. Can you provide us with the replay of that game?

Looks like that got back it in my latest commit. And it still doesn't happen on intel, so there's something at play here. I'll try changing it later.

I wouldn't use a function if I were you and just use global variables, depending on what you want to do. It's not like it's going to become a huge issue. Alternatively, you could return a tuple with all stuff in it. Or a dictionary, yes. But again, not a python expert.

python 3 recently added the int8frombytes and I find it more readable. No other reasons. Also not an expert in python, far from it. I don't understand your question overall.

Part of the problem is that our map are 16-bit grayscale, which is a bit of an odd format. Using Pillow on Python I used a 32-bit floating point texture, saved in tiff. FYI here's the bit of my script (taken as is), with the useless stuff removed: with open(os.path.join(sys.argv[1],i.replace(".xml",".pmp")), "rb") as f: f.read(4) # 4 bytes PSMP to start the file f.read(4) # 4 bytes to encode the version of the file format f.read(4) # 4 bytes for the file size patchSize = int.from_bytes(f.read(4), byteorder='little') heightmap = [[0 for x in range(patchSize*16+1)] for y in range(patchSize*16+1)] textureTile = [[0 for x in range(patchSize*16)] for y in range(patchSize*16)] # Height,qp is 16 bit integer divided by 92 to yield the real height for x in range(0, patchSize*16+1): for z in range(0, patchSize*16+1): heightmap[x][z] = int.from_bytes(f.read(2), byteorder='little') / 92 numTex = int.from_bytes(f.read(4), byteorder='little') #print(str(numTex) + " textures used.") texNames = [] for texI in range(numTex): # first 4 bytes are texture name length then read the texture name itself name = f.read(int.from_bytes(f.read(4), byteorder='little')) texNames.append(str(name,"utf-8")) #im = Image.new("L", (patchSize*16,patchSize*16) ) #pixels = im.load() for z in range(0, patchSize): for x in range(0, patchSize): for pz in range(16): for px in range(16): texA = int.from_bytes(f.read(2), byteorder='little') #pixels[x*16+px, z*16+pz] = (texA) f.read(6) #second texture (2) + priority (4), don't care #im.save(sys.argv[1] + "." + i + "output.png","png") #im = Image.new("F", (mapSize+1,mapSize+1) ) #pixels = im.load() #for x in range(0, math.floor(math.sqrt(mapSize))*16+1): # for z in range(0, math.floor(math.sqrt(mapSize))*16+1): # pixels[x,z] = heightmap[x][z] #im.save(sys.argv[1] + "." + i + "output.tiff","tiff") #EOF

If you need help with the python script, as part of my markov chains script I have a Python script to load a pmp map which could also be used to write a PMP map if needed.

Should be somewhat trivial to adapt to use in skirmish maps as the "script" part, then you could load it up, open the map on Atlas, save it, remove the script. I agree it would be a nifty Atlas tool also, but I don't think Atlas supports running arbitrary JS, so it'd have to be C++, and that would make it much more complicated to implement. As for creating a more complete map using real markov chains, it sounds possible but I think it'd require a different, more "top down" approach.