Reimplementing the game GUI with HTML

[Ykkrosh]

There's a crazy idea which I've had for a while, to replace the game's GUI system with a web browser engine. (WebKit in particular, which is used by Safari and Google Chrome). I've never got further than a very rough prototype, but I still think it's interesting, so I'll dump some ideas here and see if anyone has any relevant thoughts.

(Warning: This is mostly quite technical and I'm not explaining it well or in much detail.)

The general notion is to write the game's GUI (the pre-game menus, the in-game session UI, etc) with HTML and CSS and JS like a normal web page, rather than using a limited custom C++/XML/JS system. WebKit can be used to execute and render the pages, and the output can be copied on top of the game's OpenGL graphics. (The typical UI of a web browser (toolbars and menus and windows and tabs etc) is not relevant here - we just care about the rectangle that web pages get drawn into.)

I think the main benefit is that it would provide a lot of flexibility once it's integrated: we could use full HTML/CSS formatting to write an in-game manual describing the history of all the units; we could have an online in-game service to browse and download new maps, or to find multiplayer matches; we could add videos and fancy graphical transitions and effects. None of those are critical for a working game, but they'd be nice to have. Most are possible without a full browser engine, but they'd be much easier if we had one.

You can write pages like this (best viewed in Chrome or Safari; hover and click the 'single player' bit for effects) - view source to see that the layout and dynamic behaviour are reasonably straightforward. (Well, straightforward once you understand HTML and CSS and JS and JQuery...) Also it's easy to test using normal web browsers and web development tools (Firebug, WebKit's Inspector, etc).

Cross-platformness is a non-trivial requirement. There are lots of ports of WebKit - the original OS X one (for Safari), and a Windows one (for Safari), and Gtk (mainly Linux, sort of works on other platforms), and Qt (presumably works on all platforms), and Chromium (works on all platforms to varying extents; used by Google Chrome), and some others. We want to run on all platforms (by which I mean Windows/Linux/OS X), and don't want to write lots of platform-specific code, so we'd have to use the Qt or Chromium ports. I don't know how well the Qt port works, and I don't especially like the idea of having a dependency on Qt, so Chromium might be more interesting.

(There's already a library for embedding Chromium in (primarily) 3D games, and it's used by EVE Online, but it's no longer open source and it's not targeting Linux.)

Obvious drawbacks are that it's complex (particularly it makes the build process more painful), and might be slow, and might use lots of video memory, and adds ~20MB to the binary distribution, and maybe it's not really that useful. But I'll ignore those issues for now.

General idea for implementation: Create a shared library (.so / .dll / etc) which exposes a plain C interface, somewhat similar to the webkit_web_view_* functions in the Gtk port (which I used in my rough prototype) - create a view, navigate it to a URL, and also send mouse/keyboard events to it, and render it to a plain char* buffer (at least for the simplest initial version). The game engine would set up a view, pass events to it, then render it and copy the output into a GL texture. And it'd have some way to pass events from web page scripts back to the game engine (for "start game" and so on).

The library would be a fairly thin wrapper around Chromium's WebView and related classes. The test_shell (particularly test_shell.cc, and web{widget,view}_host_*.cc) has bits of code that use WebView, and could be informative.

A better solution would involve running the WebKit code in a separate thread, using callbacks to notify the engine when it needs to repaint or respond to unhandled input events, and only repainting (and uploading to the GPU) the parts that have changed, which might result in more acceptable runtime performance.

So I think this could be useful and feasible. But it'd involve quite a bit of complex technical work, and I have enough other things to work on myself, and ought to focus on things that are really necessary for the game, so I can't look into this as much as I'd like . But it'd be pretty neat if it worked, so I'm just throwing this idea out there

[Matei]

I think this is a great idea, Philip, sidestepping a lot of issues associated with building our own GUI that we'd have to address as we go on. (Even if we started with a very simple GUI, inevitably, people would want more from it, and would have to hack it in somehow.) Defining GUI pages using standards (HTML, CSS, etc) is also a big win for modding. The only question is whether Webkit embedding is really as easy as advertised. Seeing EVE use Webkit is very promising. I also know that a lot of apps on Mac OS use it (beyond the browser), and it claims to be designed for embedding.

[kaiserfro]

I think this is a great idea too. I like the idea of being able to develop and test the GUI without the game itself. This could attract some new developers.

Its ashamed that awesomium is closed source...as that sounds like it may be the way to go. It also sounds like the google api port may be a bit overkill. So it sounds like a new webkit or google api wrapper will be in order.

I'm not all that versed about embedding webkit in a GL-based game a a texture. I've heard of people attempting this since webkit can render to a buffer. So I have a question...In a browser the rendered page is an dynamic entity, with controls (drop downs, links, etc.), and the DOM is modifiable on the fly. If the page is rendered to a buffer, that sounds a bit static to me. How does the user interact with the rendered HTML? Its like the GL engine would need to capture mouse events and trigger re-renders of portions of the HTML page to be redisplayed on the texture.

Once thats all square, then you could do neat things like JS code to modify the pages on the fly. Maybe some AJAX like interactions for communicating with a game server and updating the pages accordingly.

[kaiserfro]

(There's already a library for embedding Chromium in (primarily) 3D games, and it's used by EVE Online, but it's no longer open source and it's not targeting Linux.)

Reading the blog of the developer, it looks like Awesomium 1.5 will be free for non-commercial use. See the blog. Unfortunately, it looks like much of this work is for windows. It may not be so portable.

[Ykkrosh]

"free for non-commercial use" sounds to me like it's not open source, so we wouldn't want to use that. But given that the last blog post was in October, in which he says he "will be announcing some of those details in the coming week", it's hard to work out what's actually going to happen. Guess we just have to wait and see

(Incidentally, this is an interesting perspective on Google's Chromium - it's not really a well-behaved system, at least on Linux, because it's provided as a monolithic lump of code (an 800MB download) with everything bundled and little attempt to integrate properly with the host system. It's less of a problem when we just want the WebKit part, not the browser UI, but it's still a problem. Other ports of WebKit seem to be much better at that.)

If the page is rendered to a buffer, that sounds a bit static to me. How does the user interact with the rendered HTML? Its like the GL engine would need to capture mouse events and trigger re-renders of portions of the HTML page to be redisplayed on the texture.

The basic idea is to render the web page to a GL buffer once per frame, so it's always up-to-date, and pass mouse events from the game back to WebKit so it can update its internal representation of the page which will be used to render the next frame. When you render, WebKit can tell you what region of the screen has actually changed since last time, so you only need to re-render that region into the GL buffer and upload it to the GPU, and if little is changing then it should be fast.

[kaiserfro]

It looks like once upon a time the Awesomium was GPL..but thats probably outdated. The new one matches up with newer versions of Chromium, but the license is not compatible.

I agree that Chromium is unstable. In fact, Google's development model is to always push ahead, since they dont really have any customers for their work, they don't need to bug fix/stabilize releases. Besides, they way they figure it, everything is in "Beta" stage..nothing ever gets finalized. Maybe things change once Chrome OS starts becoming available on netbooks.

Having said that reading about the API that chromium provides value added to webkit maybe overkill. To make the thinnest implementation, writing a new webkit binding would be ideal. A webkit to GL binding of some sort. Basically, do what awesomium does, but directly to webkit and the javascript engine, instead of through the chromium api.

Wow, what a task.

[eyecreate]

What about using this?

http://www.sirikata.com/blog/?p=115

http://github.com/sirikata/berkelium

This is a open-source clone of Awesomium that works on windows and linux. (not sure about osx)

[Ykkrosh]

Nice, I hadn't seen that before - the FAQ claims OS X support too so it covers all the interesting platforms, and it looks more focused than Awesomium on simply providing a portable embedded browser (which is what we want). This definitely seems like something worth watching

reading about the API that chromium provides value added to webkit maybe overkill

The interesting part of Chromium (from my perspective) is not the API, it's the backend - graphics engine, network layer, video codecs, database, filesystem interface, etc. WebKit itself doesn't contain any of that stuff, so you have to combine it with some usually-platform-specific port (which rely on OS X frameworks, or on Gtk, or on Qt, or (like Chromium) on some other set of libraries). The ports provide the embedding API, and Chromium also adds a (optional?) new JS engine, but the portable backend is what we most need.

[Kimball]

I may be a little late in this discussion, but I was wondering if this is still feasible. It'd make GUI development overall much easier from my perspective, and as you said, you wouldn't have to worry about rewriting everything 3 times. Also, as kaiserfro pointed out, we'd be able to test the UI without actually running the game.

[Ykkrosh]

It still could be done, and it'll become a bit easier over time because people are writing more of these WebKit-embedding libraries, but it'd still be rather a lot of work (I expect it'd take me at least a few weeks of work to get all the bits integrated and ported) and it's not really critical since we already have a mostly functional system, and I'm not even certain it'll technically work well enough for what we want (particularly in terms of rendering performance) so we'd need to spend time experimenting first. Some of the testing benefits are mitigated by our GUI now supporting hotloading, so you only have to start the game once and then it'll dynamically load and display any changes, which makes development substantially faster and less painful than before.

I still think WebKit would be more cool than the current system, but I still don't think I can justify spending my own time on it when there's so many other things to be doing (and I don't know who else would be willing and able to try it)

(On a vaguely related note, the new Steam beta client "swapped out the Internet Explorer rendering engine with WebKit, which gives us a bunch of size, stability and performance benefits" (and also gives them OS X compatibility, which it seems they're going to start supporting) - WebKit seems to be becoming pretty popular.)

[Jeru]

Guess what? Mozilla borrows from WebKit to build fast new JS engine.

[MishFTW]

Oh wow... I've been toying with this idea too. I went Googling for solutions and this came up

I think this is a great idea, Philip, sidestepping a lot of issues associated with building our own GUI that we'd have to address as we go on. (Even if we started with a very simple GUI, inevitably, people would want more from it, and would have to hack it in somehow.) Defining GUI pages using standards (HTML, CSS, etc) is also a big win for modding.

I may be a little late in this discussion, but I was wondering if this is still feasible. It'd make GUI development overall much easier from my perspective, and as you said, you wouldn't have to worry about rewriting everything 3 times.

Fast forward three years and see how far its come. If intergrated, we can use CSS media queries to create a "responsive" GUI. Plus it seems a lot easier this way.

[zoot]

Blame the filter bubble.

[zoot]

Fast forward three years and see how far its come. If intergrated, we can use CSS media queries to create a "responsive" GUI. Plus it seems a lot easier this way.

Another benefit is that there are tons of UI artists working in HTML/CSS while essentially none are working in "Pyrogenesis XML".

[Zeta1127]

XML you say? I have some experience with XML, primarily its use in modding Star Wars: Empire at War (EaW) and its expansion Forces of Corruption (FoC).

[zoot]

XML you say? I have some experience with XML, primarily its use in modding Star Wars: Empire at War (EaW) and its expansion Forces of Corruption (FoC).

Give it a shot: http://trac.wildfiregames.com/wiki/Mod_Layout#gui

[MishFTW]

@zoot, I wouldn't blame the filter bubble too much (funny thing I saw a TED talk about it a few weeks ago )

I know implementing this will be a pain but just think of the possibilites. More people grasp HTML, CSS and JS. There are things like CSS media queries and jQuery that can extend the GUI to a whole new level. Above all that, its going to much easier to modify it. The only thing I can see against a transition to Webkit is that it might break and the fact that its a ton of work.

[Gen.Kenobi]

@zoot, I wouldn't blame the filter bubble too much (funny thing I saw a TED talk about it a few weeks ago )

I know implementing this will be a pain but just think of the possibilites. More people grasp HTML, CSS and JS. There are things like CSS media queries and jQuery that can extend the GUI to a whole new level. Above all that, its going to much easier to modify it. The only thing I can see against a transition to Webkit is that it might break and the fact that its a ton of work.

The beauty of FOSS development is that you can create a branch and do your own stuff If you think you could do something like that, I really don't see why not - I mean, I guess the web guys will be more free once the website is launched

[zoot]

Another disadvantage (apart from the possible memory bloat) is the loss of fine-grained control. Consider how many websites - even today - resort to Flash when they need to do anything even remotely advanced visually. I don't think we would want that clunkiness for every effect we want to add...

[MishFTW]

@Gen. Kenobi: I intend to do something about it

Another disadvantage (apart from the possible memory bloat) is the loss of fine-grained control. Consider how many websites - even today - resort to Flash when they need to do anything even remotely advanced visually. I don't think we would want that clunkiness for every effect we want to add...

If you have the entire Webkit library with the game how would you have to resort to Flash? HTML5 and CSS3 have come a long way. With a Webkit UI scaling won't be an issue (with SVG support and media queries).

[zoot]

Does WebKit support WebGL?

[MishFTW]

Does WebKit support WebGL?

Hmm, yes it does. But as Phillip mentioned two years ago, there is Awesomium. It turns out they're free for non-commercial projects.

[zoot]

If it's not open source (GPL compatible) it's useless for us.

[Potter]

Instead of using a full fledged browser, I feel it might be better to use something like http://librocket.com/ which supports HTML and CSS, and is already integrated in many games...

[zoot]

Instead of using a full fledged browser, I feel it might be better to use something like http://librocket.com/ which supports HTML and CSS, and is already integrated in many games...

http://librocket.com/wiki/license