Mutex

[Mercury]

I'm thinking about a strategy to add more threading. Components would be protected from memory collisions by mutex*. Is there any other data that needs mutex protection? Any data which is both written to and read from during a game?

 

*incrimenting read/readwrite, at the ComponentType level of granularity

Edited August 16, 2022 by Mercury

[Stan`]

Mutexes with an event based system seems like asking for trouble. Also what about the js code ?

[Mercury]

The first phase would just be to separate simulation from graphics. This alone is worth the trouble.

I don't understand what you mean regarding mutexes in an event based system. What sorts of problems do you have in mind?

Regarding the more ambitious project of multithreading the simulation it's self the javascript is one issue to deal with, but i think not insurmountable. Some code which is currently in JS might have to be rewritten in C++, we would have to consider other engine users as well of course.  The threading model I'm thinking of for the second phase is to split into multiple threads during certain expensive tasks and then continue as a single thread soon after, so in some cases JS isn't involved at all. One option is to generate a priority queue sorted in some deterministic order (entityID maybe) multithreaded in c++ and then pass to JS. It's not an easy problem.

Edited August 16, 2022 by Mercury

[Stan`]

Rewriting JS code into C++ means more performance, but less moddability and breaking all the mods that depend on that...

7 hours ago, Mercury said:

 

I don't understand what you mean regarding mutexes in an event based system. What sorts of problems do you have in mind?

Well isn't the whole point of a message system to be asynchronous?

[Andrettin]

Parallelizing logic which is using the same data is often problematic, as even with mutex protection you might run into issues. For example, you might protect the data from read/write collisions, but the data might not be in a state that you want to display to the user instead. You could put mutices at the level of larger operations to prevent that, but then they quickly become very expensive performance-wise, possibly entirely denying any gains from parallelization.

Additionally, it is MUCH harder to see what is going on if you have multiple threads accessing the same data, making it easier to introduce bugs into the game, and harder to debug them.

Parallelization usually works better when you have well-defined inputs and outputs for what you are running on the extra threads, i.e. when they don't affect other threads while they are running, only their outputs are consumed at the end; and the input data is copied for each thread if it could otherwise change.

[phosit]

Every component could do anything. Because of that it is hard to paralelize the message-system; and it is hard for the compiler to make optimizations.
Did i understand you correctly: you want one mutex per ComponentType? That would be useless because a recource might be mutated by multiple ComponentTypes.

The function taking long is PostMessage. This function sends a message to all components which have subscribed to the messageType. Some subscribed components do nothing: RangeManager subscribes to PositionChanged but has to check if it "trackes" the entity (CCmpRangeManager.cpp L570). This could be changed to first ask all subscriped component if they realy are interested in the message (This could be done in paralell). And then sending the message to the interested components (in sequence).

An other idea: At the moment most Components have a switch in the HandleMessage. If Messages were static types this switch would not be necesary. This aproach might interfere with JS.

[Stan`]

Here is a 470ms frame in Combat Demo (Huge) You can reproduce it easily by playing that map, or by using the automated version I made here trailer_tools/maps/scenarios at main · 0ADMods/trailer_tools (github.com)

Messages here are  onownership + ondestroy. Most units don't care about it though. In theory that also gets send to all the alive entities, trees building etc.

[Mercury]

Quote

a recource might be mutated by multiple ComponentTypes

I'm not sure I understand, each access of a component (or other data we need to protect?) would require acquiring either a read lock or a read write lock. Where the access is coming from shouldn't matter.

Dynamic message subscription is on my todo list.

Added static message types to the list as well, thanks.

Anyway, regardless of the (very much non-trivial) difficulties of multitheading simulation it's self, just separating it from the main thread seems like it would give a very large amount of simulation performance when keeping graphics smooth under load. 20fps * 16ms/frame = 360ms/second: A 56% increases in simulation time budget! At least for any machine that has two or more cores (4+ threads). And that is just considering simulation lag. If we consider animation lag then each simulation turn only gets ~34ms to run in (at 20 fps). In a dedicated thread this is not an issue at all: frame rate remains constant despite simulation lag. Do we have any data on what fraction of the users have a single core machine?

Edited August 17, 2022 by Mercury

[phosit]

Scenario: two component of different type lock their mutex. But both could access a shared/global resources.
Oh... now i get the question of this whole thread: you asking about thous shared resources. I don't know but i suspect there are many. And much code to check, also JS.

We have information about the cpu's used: https://feedback.wildfiregames.com/cpu/

[Mercury]

Thanks.

Looks like maybe 2% of users using two core / two thread celeron processors. Still may work better for them to separate simulation from graphics since os doesn't take that much time* but still not great. Is there a way we can test on the very low end?

The mutex belongs to the data being accessed, not to the code which is accessing it.

*Particularly when using allocators.

EDIT: just checked and found my laptops a9-9425 is also 2 core / 2 thread, guess I can test this. Also about 1% of users are on 2 thread AMD machines.

Edited August 17, 2022 by Mercury

[Mercury]

16 hours ago, Stan` said:

Rewriting JS code into C++ means more performance, but less moddability and breaking all the mods that depend on that...

Well isn't the whole point of a message system to be asynchronous?

In an extreme case we could have C++ which could optionally be replaced by js. Probably not needed. I hope.

The purpose of a mutex is to prevent memory collisions: when threads try to write or read memory in the middle of another thread writing the memory. Regardless of the details of where threads split and join the mutex is needed, even just to prevent the main thread reading data while the simulation thread is writing it.

Edited August 17, 2022 by Mercury

[Grapjas]

2 hours ago, phosit said:

https://feedback.wildfiregames.com/cpu/

Weird. I have feedback enabled but mine isn't in the list (Ryzen 5700x).

EDIT: Ah nvm, must be under other.

Edited August 17, 2022 by Grapjas

[Stan`]

8 hours ago, Mercury said:

In an extreme case we could have C++ which could optionally be replaced by js. Probably not needed. I hope.

I think removing moddabilty will kill the game Because as that point going with a commercial engine might be smarter ^^

You can share interfaces for components. See UnitMotion and UnitMotionFlying which share the same interface. One is JS the other C++

But IMHO if we thread the sim ai and gui which would be good (and all have to be in the same thread cause js contexts are not thread safe)  it should not require components to be changed.

 

[alre]

10 hours ago, Mercury said:

In an extreme case we could have C++ which could optionally be replaced by js. Probably not needed. I hope.

why not consider this? is there really any downside to convert a component from js to c++?

[Stan`]

16 minutes ago, alre said:

why not consider this? is there really any downside to convert a component from js to c++?

Not moddable anymore, breaking all mods changing it. If not done properly overlflows and out of sync

[alre]

1 hour ago, Stan` said:

Not moddable anymore, breaking all mods changing it. If not done properly overlflows and out of sync

does your answer consider the quote in my question? 

[Stan`]

38 minutes ago, alre said:

does your answer consider the quote in my question? 

My bad, don't think one can easily overload C++ functions, the roundtrip between js and c++ could be more costly than just js. Also you cannot extend interfaces with more functions, nor can you change the C++ code that use it. Then you have to overwrite those C++ functions with js ones too

I suppose it should be profiled. 

[Mercury]

11 hours ago, Stan` said:

My bad, don't think one can easily overload C++ functions, the roundtrip between js and c++ could be more costly than just js. Also you cannot extend interfaces with more functions, nor can you change the C++ code that use it. Then you have to overwrite those C++ functions with js ones too.

We wouldn't need to switch to js just to check for existence of a file. Also the result can be cached.

Regarding the initial point of the thread though: today I'm thinking mutex is overkill and we can use std::atomic. Just need to figure out what all data needs to be marked atomic.

Edited August 18, 2022 by Mercury

[Mercury]

Some further thoughts: There is no possibility of deadlock if only one thread at a time acquires a lock while holding another. A global JS mutex can designate that thread.

When using mutex the primary danger is deadlock: when one thread holds mutex A while attempting to lock mutex B while another holds B when attempting to lock A.

JS needs a global mutex and js code will need to lock arbitrary compiled-side data.

So we can't prevent the thread which holds the JS mutex from locking additional mutex concurrently, creating half of a deadlock.

What we can do is not provide the other half. The compiled code could be modified so all shared data is protected by mutex with the rule that nested locks are prohibited. If in some parts of the code it is not possible to do one mutex at a time without exposing an object with incomplete state we can use std::lock to lock multiple mutex atomicly. The resulting code will I think not be too complicated and the rule of no nested mutex is relatively easy to verify. As long as only the thread which currently holds the JS mutex is allowed to do nested mutex locks there is no possibility of deadlock.

 

EDIT: I forgot to mention, this plan also requires a message queue, to reduce nested mutex instances which will need to be addressed to direct access via CmpPtr. All Post and Broadcast can be made safe automatically.

Edited September 5, 2022 by Mercury