Ykkrosh

Apparently it works when using SuperPNG for loading (not just saving), so I think PNG is okay now?

It already handles both (as long as you disable RLE compression of TGAs), so there's no technical restriction as far as the engine is concerned. It's more of a logistical problem - it's good to stick consistently with one format, it's good to reduce SVN disk space usage, it's good to be compatible with all significant paint programs, etc. PNG seems to match all of those requirements, except for compatibility with Photoshop (Did you see the link I had to the SuperPNG plugin? If that works then hopefully there wouldn't be any other problems with PNG.)

I don't believe we have any firm plans for single-player yet, so it's not too late to discuss ideas for this Personally I never play multiplayer or skirmishes in RTS games, because I want games to give me a sense of artificial achievement without making me work hard - multiplayer involves too much losing and skirmishes are pointless, so I usually just play the campaigns. I think it'd be good if we had a way to engage players like me without the expense of creating extensive campaigns (though I'd like those too), by building some meaningful framework around skirmish matches.

Yeah, I didn't realise it'd be a problem, but people say "It's frustrating because back around version 5.0 or 5.5 Photoshop would actually save the full channel on PNGs. Adobe changed the way they handled them for subsequent versions of the software." and everything I've seen agrees with that. I've not seen any explanation of why it was changed.

GIMP seems to do the same for alpha channels in DDS and PNG and TGA, so the choice doesn't make a difference there. (Looks like it handles channels automatically, or you can use the Layers thing to add a Layer Mask based on the alpha channel and then delete the alpha channel, which lets you easily view and edit the colour/alpha independently.) It's still important to support Photoshop too - if that SuperPNG plugin works then maybe that's okay, otherwise we could maybe do something like store PNGs in SVN (since they're smaller and should work for most modders) and have a tool to convert to/from TGA for use with Photoshop (assuming it handles TGA better than PNG).

Hrrm, it sounds like Photoshop is weird and stupid. What version are you using? There's some discussion here with people having similar difficulties, with a link to a SuperPNG plugin that apparently might help. If it's troublesome, would TGA work better?

Yes - it should work the same as with TGA or uncompressed DDS. (Those formats will still work too, PNG is just preferable since it reduces SVN download size.)

I've committed a hopefully-working version of this now. If people want to check some of the changes to find problems or design errors, please do The first time you run the game, everything will look grey while it loads and converts the textures. They'll get cached so subsequent loads should be instant. The cache is stored in %appdata%\0ad\cache\ on Windows, and ~/.cache/0ad/ on Linux - you can delete it if you want to free some space (it shouldn't get more than a few dozen megabytes) or watch the exciting texture-conversion process again. Only SVN users and modders should see this conversion process - players of releases will get given pre-converted textures, at least once I've got around to implementing that (before the next alpha release). There's some documentation on the wiki for artists/modders about how to use this system.

Just to check, did you make all the components of the image yourself or get them from freely-reusable sources, or did any come from non-free sources? (If we use it in the game we need to be reasonably careful about copyright licensing.)

The resources are still food/wood/stone/metal as far as the player is concerned - the implementation was just changed to have sub-types for them all. You can get food from wildlife and fish and farms, you can get metal from ore and from treasure chests or whatever, etc. Animations are now identified with those sub-types (since they might be different for each), not with the generic types.

I don't see any fundamental reason why we couldn't do that - it'll just need some coding and maybe some new models. Sounds like it would be a good design to copy from.

You need to use ctrl + middle-mouse-drag (or right-mouse-plus-middle-mouse-drag) to rotate the camera in Atlas - see documentation. See the next section there for rotating buildings, or click-and-drag when first placing the building. Resizing maps currently isn't possible, except by opening binaries/data/mods/public/maps/scenarios/_default.xml in a text editor and changing the patches="12" value. There isn't any elevation picking tool yet (though it probably wouldn't be hard to add). You can elevate a point by just changing the brush size to be very small, but I'm not sure if that's what you mean. Units shouldn't be able to walk through walls - I think their collision shapes are just set wrong. I believe you can get a unit to stand on a spike of terrain inside a wall, but the pathfinder will probably get quite confused if it tries walking anywhere.

See the build instructions - you need a C++ compiler, and probably a reasonable level of C++ experience . Currently there's no way you can communicate with the game from a browser, since they use completely different network protocols.

You probably need to select the "install source code" option when running the installer, then it should put all those files in the install directory.

The networking source code is in source/network/. Currently it uses UDP port 20595 for client/server communication.

I've thought about embedding WebKit, but I think it'll be quite a pain to do it portably and efficiently. It'd probably be easier for an in-game lobby client to just use our current custom GUI system for rendering (and we should add an HTTP API so GUI scripts can communicate with the server).

Thanks for discussing this - I think I'm getting closer to understanding it now I'm setting it up to convert all textures into DDS, even when the original is DDS - that's useful since the converter can regenerate mipmaps, making sure they always exist (so we don't need any runtime mipmap generation) and using sharper filtering and gamma correction. NVTT requires BGRA input so I'm just using our existing texture loading and transform code. But after some time I noticed that the converted textures were very slightly greener than the originals - it turns out that our S3TC decoder's unpack_to_8 is bogus, but unfortunately not bogus enough that anybody noticed the graphical problems before now . I think I've fixed that (and added some basic tests), so the output seems pixel-perfect now. The nice thing is that texture-quality bugs shouldn't be a major concern - we can just change the code or compression settings and it'll automatically reconvert everything with no hassle.

Hmm, I guess there's a few ways we could support external lobby clients, but I'm not sure what would be best.One option is to require that the user already has the game open and running. The game could run a local HTTP server on some predetermined port, then the lobby client web page could do <iframe src="http://localhost:12345/0ad-rpc.html"></iframe> to open a connection to the game and use postMessage() to do bidirectional communication. That could be interesting for more than just lobby clients - it could provide an API for writing out-of-process AI players, for collecting performance stats, for running automated tests, etc. Another option is for the game to register a protocol handler, so the lobby client can navigate to a URL like "zeroad://connect?ip=12.34.56.78" which will launch the game even if it's not already running. Need to be extremely careful about security, though - any random malicious web site could trigger it at any time, and safely parsing the command-line options doesn't sound easy (it seems a common security vulnerability). (Is it possible to do similar things with MIME types rather than protocol schemes? I guess it's somewhat platform-dependent.)

(Threaded compression is fun - everything is nice and smooth (at least on dual core) and the textures slowly but magically pop into existence. Writing threadsafe code is hard, though - I don't think I can even use WriteBuffer since it can call io_Allocate which can use the global AllocatorChecker, which is unsafe by itself and also calls debug_assert which is unsafe. But I think in this case I can allocate the buffer in advance so it's not too bad.) Sync loads are needed about once in the whole engine, so they're not very interesting. Prefetching won't call Bind (since the whole point of prefetching is to load textures before the renderer first needs to bind them and render them). So Bind will always be associated with the load-synchronously-unless-it's-going-to-take-too-long behaviour. Not reliably - e.g. in the sync loading case of Atlas wanting to draw terrain previews, the same terrain textures may or may not have already been loaded by the normal renderer or the prefetcher, so it may or may not give the warning. In the prefetch case, the renderer can't assume the prefetcher is perfect and anyway it'll want to boost the priority of the texture it's trying to render, so it'll always have to call the TryToLoadNowIfItsNotTooMuchBother() function before Bind() anyway, and we wouldn't get warnings about forgotten prefetching.

(Hmm, compression is irritatingly slow - I'll probably see if I can move just that part into a thread for now (but not any of the IO code or anything), so the game stays playable while it's busy compressing (e.g. when you change the compression settings file and it hotloads all the textures).) Ugh, that sounds surprising (i.e. bad) and could cause unnoticed weird behavior (e.g. if callers actually wanted sync loads, or only wanted prefetching+placeholder). Why not just assert() that callers have called one of the three APIs before Bind()? It doesn't seem that surprising to me - these are load-on-demand texture objects, and Bind() is the demand, so it should load them. Requiring an explicit load call would add (a small amount of) complexity to the renderer logic, and assert failures wouldn't be trivial to debug (they might occur in rarely-used codepaths, and if a texture is shared by two different systems then it might depend randomly on the order in which those systems run), so it doesn't sound like an improvement.

Uh, I thought the main point of doing this stuff asynchronously was to avoid variation in latency and keep a consistent framerate even when loading. Optimising just the worst case (dying hard disks etc), at the expense of the best/typical case, sounds counterproductive.Maybe we could do both if it was multithreaded, though: LoadTexture can start a background load, wait up to 0.5 sec for a reply, then fall back to the placeholder texture if it takes longer than that, so it'll never freeze for long. Hmm... Another quick measurement: with cold cache, loading and uploading terrain/model (not GUI) textures for the initial view on Arcadia takes 2 secs; CObjectEntry::BuildVariation (which I think loads prop actors and meshes and idle animations) cumulatively takes 8 secs. (With hot cache it's about 0.1 secs / 0.2 secs). Makes sense. (Also I should probably give each texture a pointer to the texture manager, so I can just call m_Texture->Load(). And then make Bind() implicitly do the StartLoading thing, so callers don't have to remember to load it first.)

(Networking ought to be handled in a separate thread anyway, so that the communication latency isn't hurt by the framerate. I think that shouldn't be too hard to add to the current network code, except for some issues with needing to deserialize JS objects.) How are the graphics paused, given that we're already halfway through the rendering when we tell the loader we need a new texture right now? Continue rendering as normal and just skip the SDL_GL_SwapBuffers() at the end, perhaps? (I have no idea if that would really work - sounds a bit dodgy to completely stop repainting the screen. Also it means the best case would be a 1 frame delay (perhaps 50ms), even if the texture would load in under 1ms (which it does with a hot disk cache).) It shouldn't be hard to e.g. have the release process pack a 16x16 version of every texture in the whole game into a single 0.5MB texture that's loaded at startup, if we want an immediate low-detail version. Might look okay for terrain and object textures, though I guess it would be uglier for large GUI textures.Textures aren't the whole problem, though - actors and meshes and animations take time to load (much more? much less? I don't have any measurements now), and it'd be nice to handle them similarly, and there isn't any quick low-detail version we could swap in for them. So we need to handle them with some combination of prefetching and render-delaying, and fancy placeholders just for textures probably wouldn't add much benefit. Disagree - only one of several terrain biomes would be active at a time [...]By "world" I meant the currently loaded map, i.e. just the data that would be required if you zoomed out to see everything at once or if you scrolled across the whole map. I agree . I think my current rough API might be asynchronous enough already, so hopefully it's okay and we can relatively easily change the implementation later if needed. Currently it's like:// Prepare a texture (but don't do any loading yet - we might not ever want to render this texture): CTextureProperties texture(pathname); texture.SetWrap(GL_REPEAT); texture.SetMaxAnisotropy(2.0f); CTexturePtr m_Texture = g_Renderer.GetTextureManager().CreateTexture(texture); ... // Use it in the renderer: g_Renderer.GetTextureManager().LoadTexture(m_Texture, CTextureManager::PRI_HIGH); m_Texture->Bind(); glDrawElements(...); ... // Maybe we want to prefetch it earlier: g_Renderer.GetTextureManager().LoadTexture(m_Texture, CTextureManager::PRI_LOW); There's three load priorities: IMMEDIATE = it really must be loaded now, so the bind will succeed and we can read pixel data out of it. (Should be used very rarely.) HIGH = if there's a cached compressed version then load that immediately before returning; otherwise add to a compression queue, and use the black placeholder for Bind() for now. LOW = add to a loading queue. Then the main loop asks the texture manager to process some of its queues each frame, and it'll do some loading and compressing and update the Handle stored behind the CTexturePtr. The rendering code never accesses the Handle directly, so the texture manager can easily update the textures whenever it fancies (for async loading, hotloading, etc). The caller can't make any assumptions about synchrony (except for IMMEDIATE) so it doesn't restrict the implementation. Is this the kind of thing you are suggesting, or are there serious problems with this?

For normal players (who will have pre-converted cache files), I think rendering the placeholder texture (currently solid black) for a few frames would be ugly and disturbing, and worse than just pausing for a fraction of a second. We don't know every single texture that a frame will use until we actually render that frame, so the renderer will have to ask for the texture then synchronously wait if it's not loaded yet. To fix the performance impact of loading new stuff, I think it's better to focus on prefetching. Players are very likely to see almost all the terrain and all the units in the world, by the end of a game, so we'll have to load it eventually and design it to fit in their memory. So we might as well load it earlier - the first frame can render as quickly as it does now, and then there'll be some background activity for a while as it's prefetching the rest, and then it should be perfectly smooth when you scroll around. (Prefetching involves looping over all terrain tiles, and all entities in the world, and peeking in the command queue for new entities that are about to be trained, etc, and pushing all the needed resources onto a queue (and maybe doing part of the processing in a background thread) and spending up to n msecs per frame loading them into the main thread). Then it should be very rare for the renderer to need a resource that hasn't been loaded yet, so it doesn't matter that that's synchronous, and in those rare cases it'll just pause for a tiny bit instead of rendering something incorrect. (There's some special cases like the terrain previews in Atlas which need to extract the texture data after it's loaded, and making that asynchronous would be nice but probably a lot of complexity since it's crossing threads and DLLs and languages (C++ to JS) and I'd rather not bother yet. So there should be a synchronous API at least for that case.) Is it feasible to load files and textures in a background thread with the current lib code? I thought that was all non-threadsafe and probably very hard to fix. We could do the texture compression in a thread but that wouldn't be used by normal players and doesn't seem worth the complexity.

Starting to implement some of this now. Current plan is to add a new texture manager, which can construct a texture object given a filename and set of texture parameters (wrap modes, etc), and then can perform the actual loading (possibly involving slow conversion/compression/packing/caching based on the source textures plus some metadata files) either synchronously or asynchronously (not multithreaded, just spreading the set of textures over multiple frames). The asynchronous bit means you won't be blocked for a minute when first starting on an empty cache - it can display a placeholder and let you test the game while it's doing the compression in small bursts between frames. Hopefully that'll reduce the biggest pain with this approach.

You need to run "make clean" and run update-workspaces.sh again, after updating from SVN, if any source files have been added or deleted recently (which they have). That will fix the "No rule to make target" errors. (I expect you'll still get the "invalid conversion from ‘volatile int64_t*’ to ‘volatile intptr_t*’" errors, though.)