Architecture for new sound system

[k776]

Hey all,

Lets discuss the architecture for the new sound system. The following is what I have so far. When we have a solid plan, I'll update http://trac.wildfiregames.com/ticket/1223

New Sound System

We need a new sound system, that works on Windows, Linux and Mac. Optionally also works on Android systems, so design with cross-device in mind (wrappers, apis etc), but focus on the operating systems first before mobile platforms.

The new sound system needs support for music and sound clips, and should work on both the main menu and ingame. The same system should handle things like background music, button clicks, ambient noise etc.

The implementation should be 100% C++ object oriented, but should have JS interfaces so that GUI events can trigger sounds (like button presses).

Details

Terminology: 'stack' (collection of 'tracks' on a 'queue'), 'track' (object linked to the 'target' sound to be played)

The sound system should support multiple named stacks. A new track can be added to any named stack. Stacks are worked through concurrently (multi-threaded?), playing through their tracks first in order of priority, then the order the track was added to the queue.

If a track of high priority comes into the stack, the current sound should fade out, and the priority one should fade in, then fade back to the one it was playing (requires the system have a 'pause' for tracks so that they can resume from the spot they were stopped, rather than restarting). More on track fading later.

A stack can be unlimited in size, or can have a limit. Default is unlimited. If a limit is set, when a new track is added and the limit of the stack is reached and a track of lower priority doesn't exist to replace, the stack simply discards the track that was added. This prevent many async sounds piling up and running all at once (like 100 soldiers moving, which should not create a giant "KA-THUNK" when they move).

If a stack has 'async' set to false, the stack waits until each tracks target finishes playing before popping the track off the queue and initiating the next track (useful for music). If the stack has 'async' set to true, the tracks target starts playing and the stack pops the track right away, and moves onto the next track. Default is async true.

For music in particular, we don't want it stopping suddenly and starting the next one instantly. So if a track has a fadeIn value, this is the amount of milliseconds that a volume should go from 0% to 100%. If another track is playing, the same time is used to fadeOut the playing track. The default is no fading (for things like quick unit actions, button clicks, etc).

TODO: Describe fade between tracks, Describe gain adjustment on zooming, Describe gain adjustment on distance from center screen

Basic JSON representation

{
  "actions": {
    "limit": 10,
    "async": true,
    "tracks": [
      { "target": "chop.wav" },
      { "target": "mine.wav" },
      { "target": "chop.wav" },
      { "target": "chop.wav" },
      { "target": "mine.wav" }
    ]
  },
  "ambient": {
    "limit": 10,
    "async": true,
    "tracks": [
      { "target": "tree_falls.wav" },
      { "target": "fish_splash.wav" },
      { "target": "tree_falls.wav" },
      { "target": "tree_falls.wav" }
    ]
  },
  "music": {
    "limit": null,
    "async": false,
    "tracks": [
      { "target": "peace1.wav", "priority": 2, fadeIn: 1500 }, # this would play second
      { "target": "peace2.wav", "priority": 2, fadeIn: 1500 }, # this would play third
      { "target": "war1.wav", "priority": 1, fadeIn: 750 }    # this would play first
    ]
  }
}

[feneur]

I don't have enough technical knowledge to add anything of value, however, I want to add that there was a lot of old discussions and decisions and while there might be better ways to do things/new libraries available etc etc, it's good to keep in mind what has been done (to avoid doing the same things again if nothing else). http://trac.wildfiregames.com/wiki/TDD_Audio is probably the most important since that is the end result of the previous discussions. Also Carsten's old blog about the sound development: http://www.waste.org/lostchocolatelab/0adblog.html

[jdm]

"If another track is playing, the same time is used to fadeOut the playing track."

I assume this means cross-fade instead of fade out, followed by fade in?

Edited March 20, 2012 by jdm

[jdm]

Also, how does this interact with positional sound? The ambient and action sounds presumably are only valid for a particular area of the map at a given point in time; it feels like there should be an optional link to an entity from which a position is obtained and the gain adjusted based on distance from the camera.

Also missing: default priority information.

Edited March 20, 2012 by jdm

[jdm]

Priority is amorphous in relation to async stacks, or perhaps just underspecified. Sync stacks obviously only play a single sound at a time, but async stacks play multiple ones. For an async stack, does a new sound of higher priority cause all existing lower priority sounds to be interrupted, or does the priority only affect its insertion position and all sounds in the stack are mixed together? If the former, it should be specified that only the sounds of the highest priority in a stack are played concurrently in an async stack.

I also think that multiple (or single, or concurrent) would be a clearer name than async.

[jdm]

23:40 <jdm> k776: do we need to consider looping sounds?

23:40 <jdm> having never heard the sound in 0ad, I'm just guessing here, but ambient sounds typically loop

23:42 <k776> jdm: Good point

23:42 <jdm> so, here's a strawman for dealing with ambient sounds:

23:43 <jdm> the ambient stack contains all possible ambient sounds that could ever be used

23:43 <jdm> the game loop investigates the terrain within a certain distance of the camera

23:43 <jdm> marks the appropriate ambient sounds as active

23:44 <jdm> and some special code notes sounds that are active that were not previous

23:44 <jdm> *previously

23:44 <jdm> and fades them in, and fades out sounds that are no longer active but used to be

This could be implemented as part of a general ambient sound API that is exposed. Maybe it could exist within the proposed system by manipulating the priorities of the sounds in the stack and adding a fadeIn value to each sound in the ambient stack.

[wraitii]

I believe to do this efficiently you'd need three classes inheriting from a "basic sound manager"...

-A "music" class would deal with musics... It would support pauses, fade ins, fade outs, and presumably only use one "Stack".

-An "ambient" class which would perhaps have an "ambient" sound looping, and occasional animal sounds or things like that (2/3 stacks at most)

-An "effect" class which would deal with short effects: GUI sounds, unit sounds, … They should be played immediately or not at all. I dunno how many "stacks" you'd need for that, probably a few to support simultaneous effects.

[memmaker650]

Nice to ear about new sound system.

I will re-donate some money if you need to contract a developer for the new sound system.

Great work. The game development go in the right direction.

[Ykkrosh]

I want to add that there was a lot of old discussions and decisions and while there might be better ways to do things/new libraries available etc etc, it's good to keep in mind what has been done (to avoid doing the same things again if nothing else)

Yeah, I think it's important to consider the problems and limitations with the current approach before jumping into a new one. Some random thoughts on the current state of things and how I imagine it could be improved:

Currently music is handled via hacky GUI JS code, which is the wrong place for it - music needs to persist (and smoothly crossfade etc) across multiple GUI pages, so it shouldn't be controlled by code that's inside a GUI page. It doesn't need to do anything very fancy - play menu music, or victory/defeat music, or civ-dependent session music, and switch the session music to combat/peace as appropriate, and that's probably about all. Ideally it would smoothly crossfade even if the game's main thread is stalled (e.g. while loading a map).

Currently ambient sound (birdsong etc) is handled the same way as music (i.e. badly). It might make sense to continue doing that, with a sensibly-designed music+ambience manager. (Music and ambient sounds are both long and ought to be streamed from the in-memory Vorbis content, whereas standard sound effects can be fully decompressed in advance.)

Currently we map the 3D world-space positions of the camera and units directly into OpenAL's 3D positional audio. That's not appropriate for an RTS-style camera - I believe (though not based on any evidence or testing) players expect to hear what's visible on the screen, rather than expecting to hear what would be physically realistic if they were floating in a hot air balloon above the battlefield. A unit in the middle of the screen (nearest the camera) is no more visible to the player than a unit that's near the edge of the screen (further from the camera in 3D), but those are far more visible than a unit that's just past the edge of the screen (only a little further from the camera), so a sound's volume should intuitively depend on 2D distance from screen edges rather than on 3D distance from camera.

It's still useful to hear some sounds that are outside the screen, e.g. combat sounds (telling the player that they should pan in that direction because there's a battle going on that they might have missed), while others are unwanted distractions, e.g. cows mooing (a nice ambient effect but pointless if you can't actually see the cow); so different sounds should have different falloff behaviour.

When zooming out, it'd probably make sense to make all the units somewhat quieter (to compensate for there being more visible at once and for them looking smaller) and maybe increase the volume of ambient wind etc.

The precise details likely need endless tweaking, but I think the important basic principle is that the sound designers select screen-based volume falloff (not 3D distance-based falloff), and that the engine applies some arbitrary transformation from the original 3D positions and camera and zoom and screen size etc to produce the final 3D positions that are passed to OpenAL.

Currently the engine can only trigger a sound to play immediately and independently. That makes the intensity concept infeasible to implement well - we can't detect there's a lot of combat sounds and replace some of them with a single high-intensity battle sample, because the sound system will have already started playing all the individual combat sounds, and because the sound system isn't made aware of gameplay concepts like "combat" (all it knows about is the individual sound groups). (I think the intensity concept is valuable since it lets the sound designers make large battles sound good, and would also help when formations of units are given commands (you can't just repeat one "yes sir" speech sample for every unit because you get nasty phasing effects, it'd be better to have a pre-mixed large-number-of-people-saying-the-same-thing sample), so it should be worth supporting it properly).

It's often feasible for the engine (and/or gameplay scripts) to tell the sound system that an effect should be played, some hundreds of milliseconds before it's needed. (E.g. it knows a unit is in the 'attack' state so it can calculate when the animation will reach the point that triggers the sound, assuming nothing unexpected (e.g. death) occurs in the meantime)). Also it could pass extra data to the sound system, e.g. adding a "combat" tag to the sound. Hopefully that would be enough for the sound system to build up a global picture of everything that's playing now or that will start soon, and if the total combat sounds exceed some intensity threshold then it can disable some of the not-yet-played ones and fade in the high-intensity battle sample, or something like that. Again I haven't thought through the precise details, so this could be totally wrong or overcomplicated, but I think the important general idea is to have a sound manager that isn't entirely short-sighted like the current implementation and that can analyse the whole soundscape.

Currently I don't think there's a distinction between music, ambience, effects, etc. All sounds should be classified into a small number of groups so players can adjust the volume of each group individually (e.g. if they hate our music, or if the ambience distracts from the important gameplay effects).

Currently the behaviour of sounds is defined in sound group XML files. Those files are pretty verbose and ugly and low-level - I think the general principle (of a sound group defining all the playback parameters and listing the individual .ogg files to randomly pick from) is probably fine, but the format should be cleaned up (and probably use some kind of inheritance system (as with entities and textures etc) so you don't have to repeat common parameters).

[stwf]

Hello All, As discussed in IRC I've been doing some research into OpenAL etc. To those means I've whipped up a sample app for playing sounds. The main classes I think could be a starting point to developing a new sound manager for 0AD.

Right now it consists fo a SoundManager that can create SoundItems (wav files loaded into one buffer) and MusicItems (ogg files streamed in pieces)

Sounds can be looped, played once and held, or played once and deleted. Music can loop or not.

Plans are to add:

1) caching and reference counting for the sound data so there is only one copy of duplicated sounds.

2) crossfade both slow and fast for the music

3) load music files in a separate thread.

4) Smarts to eliminate crossfading to the same song.

So please excuse my rusty cross platform C++, I know the code needs some work to be brought up to speed as well as fit in with 0AD conventions. But I think its a good starting point and can be worked on while coming up with a next-gen API that will let the game support all of the features listed here.

Its an XCode project and the code can be grabbed or browsed at:

git@github.com:stwf/SoundTest.git

I'll hang around in IRC when I can if anyone wants to discuss this.

steve

[stwf]

Hello All, I just figure I'd kick this thread ahead a little. I know this stuff isn't the most interesting to think about hence the situation! Anyway it seems like there are two schools of thought here, and I don't think theres much point in pressing on until a direction is chosen. I really don't know enough about the game to give much advice here. I also have no real preference personally, whichever would work better.

1) is to let the Javascript control everything, in which case the sound manager will just be a generic javascript interface to OpenAL suited to our means. In that case we need to just start designing the interface language. I believe that was the idea with the stacks implementation that started this thread.

2) Is to put the SoundManager pretty much in charge of things. The C code could play background music depending on the current game condition, and also keep track of the objects in the screen area and play sounds for what ever activity they were currently taking part in. Access to set these items could be extended to javascript on an as needed basis.

The third class of sounds I see as distance-proof. Like an alert for aggression or something that you want to hear no matter how far away it is, but it does have a location. This could of course be called from javascript, and maybe monitored by the Sound manager too, where ever it's feasible. Is there a need for a set of sounds with no general location, which should be heard no matter where the screen is?

Also I think the model for 3 dimensional sound is not with the listener floating in a hot air balloon, but as it would be heard on the ground in the center of the screen. Since you're usually looking at the center those things would be the loudest, and stuff happening on the right of the screen would be on the right and a bit softer. You could combine this with a sharp rolloff for things that are off the screen. Only something huge and barely off screen would be audible.

OK, discuss! Or tell me how off base my analysis is...

[feneur]

Hello All, I just figure I'd kick this thread ahead a little. I know this stuff isn't the most interesting to think about hence the situation!

It's also a bit above the heads of us non-programmers for most of the time =) Good to see you take an active interest in this though, it's sorely needed

Is there a need for a set of sounds with no general location, which should be heard no matter where the screen is?

Interface sounds would fall into this category, or am I missing something? Music should also be heard all the time, perhaps with pauses at times to give you a chance to hear something else for a while, but not limited by location I mean. Or perhaps you're talking about sounds "from the game world"?

Also I think the model for 3 dimensional sound is not with the listener floating in a hot air balloon, but as it would be heard on the ground in the center of the screen. Since you're usually looking at the center those things would be the loudest, and stuff happening on the right of the screen would be on the right and a bit softer. You could combine this with a sharp rolloff for things that are off the screen. Only something huge and barely off screen would be audible.

At least in theory that sounds great and logical, hard to know for sure without having experienced what it's like in-game though =)

[Shield Bearer]

At least in theory that sounds great and logical, hard to know for sure without having experienced what it's like in-game though =)

Anno 1404 has a great sound system(yes, everything about it is great) Which is somewhat like how stwf described. You can hear everything on screen and it gets louder the more you're zoomed in. Stuff like battles and cannons can be heard if they are a bit out of the screen area etc. Maybe download the demo and see for yourself? Not only the sound system, but the whole game

[Mythos_Ruler]

I'd actually like to hear gusts and wisps of wind start to overtake all the other sounds when you zoom all the way out. That would be great (especially for Strategic Campaigns where you are zoomed way far out). It doesn't have to be 100% accurate from a 3D "sound simulation" point of view, but should feel "authentic." There's a difference I think between an authentic experience and an accurate experience. For a first person shooter, you'd probably want to err on the side of accuracy. But for an RTS, something authentic with a good feel for gameplay considerations, colored with good aesthetics, is the way to go.

[Pedro Falcão]

"Unit responses" should have a controller to turn on/off whenever the player wants and may have only two volumes: inside the screen and far away (for when you call a unit for this point X of the map, i.e.).

Interface sounds, like mouse clicks, hotkey/button activation, ability activation are the most repetitive sounds, so they should have the best quality possible and be carefully "boring-tested".

Music are the sounds the player wants to be repeated less and should have a turn on/off button too.

Ambient sounds are the most "invisible" sounds, the player doesn't actually listen them, but with bad ambience, the player feels the artificially of the game.

Music, Ambience, Effects (this one includes both interface and responses) must have their own volume manager and a global volume manager.

[BenYounix]

Hi all,my name is Younes Benmoussa I am a student in engineering in software development and I want to contribute to this wonderful game that will have great success in the future.I often develops games as a fan of video games.

I mainly use open source APIs Such as OpenAL, OpenGL, SDL etc ...I wrote several libraries for my game engine (Sound engine, renderer engine, file manager, ...) unfortunately I have no time to continue ... its a lot of work for one person.

I would be happy to assist you in developing the new sound system. i attached my sound engine to this post so you can see my work.

for the sound engine:

- I use libsndfile to read / write sound files (very easy to use) and OpenAL library.

- i use caching for sound data in a linked list to avoid duplicating sounds.

- i play sounds in different threads.

Like i said i will be happy to give some help.

ps: im french so sorry if google translator make mistakes .

MPGE_Sound.zip

Edited April 18, 2012 by BenYounix

[stwf]

Hi

THere has already been some work done on a SoundManager, but I'll take a look at your code. See what can be used. But in the meantime I am sure the developers would welcome your help in many areas.

Maybe you should go on IRC in the #0ad-dev chatroom on Quakenet.org and see what they might have in mind.

Hi all,my name is Younes Benmoussa I am a student in engineering in software development and I want to contribute to this wonderful game that will have great success in the future.I often develops games as a fan of video games.

I mainly use open source APIs Such as OpenAL, OpenGL, SDL etc ...I wrote several libraries for my game engine (Sound engine, renderer engine, file manager, ...) unfortunately I have no time to continue ... its a lot of work for one person.

I would be happy to assist you in developing the new sound system. i attached my sound engine to this post so you can see my work.

for the sound engine:

- I use libsndfile to read / write sound files (very easy to use) and OpenAL library.

- i use caching for sound data in a linked list to avoid duplicating sounds.

- i play sounds in different threads.

Like i said i will be happy to give some help.

ps: im french so sorry if google translator make mistakes .