FreeType - TTF fonts in OpenGL

[RedFox]

A few weeks ago Michael (Mythos_Ruler) mentioned his problem with current 0 A.D. bitmap fonts - they seem to fall quite short from rest of the art content, mainly because:

1. Texture fonts aren't scalable (you need to generate fixed sizes).
   
2. No dynamic font stroke, outlines or drop-shadow effects (unless baked into the texture).
   
3. No wide Unicode support (cyrillic, greek, extended latin, arabic, etc.).
   
4. Small text is quite hard to read (no font hinting).
   

I've been working on this problem for the past two weeks and decided to give you some insight to the work.

1. Old bitmap fonts

The old bitmap-font system in 0 A.D. is very simple, yet limited. A bitmap font is usually generated with an external tool and contains glyph information (*.fnt) and the bitmap (*.png). To illustrate, here is the current 0 A.D. 'Serif 12px' bitmap:

You can immediately notice that most characters looks somewhat pixelated and dirty, making some characters pretty hard to read. This is mostly due to imperfect anti-aliasing for small scale fonts and no font hinting.

Adding new fonts is especially difficult, since you need to generate font bitmaps for any variation you want. Say you wish to have Arial text in sizes 10, 12 and 14; you also might want regular, italic, bold, bolditalic styles. You also might want to have a regular, stroked and drop-shadow variant.

If you had to generate all these bitmap fonts, you'd quickly have [sizevariants] * [stylevariants] * [fxvariants] font variants. In this case it might be 3 * 4 * 3 = 36 variants. So now adding a new font into the game with all the features you want turns out to be a real hassle since you need to generate so many variants that its crazy. Running tests to find your 'favorite' font and style combination quickly becomes pretty much impossible. Once you've generated your Arial variants, you'll just be glad its over.

Furthermore, if 0 A.D. ever wishes to become a fully localized / multi-lingual game, at least partial unicode support is required to draw cyrillic, greek, extended latin, arabic, etc. glyphs. Can you imagine generating bitmap fonts for such a broad variety of languages? Add 6 different codepage support and you'll be looking at 36*6 = 216 bitmap fonts.

Currently 0 A.D. bitmap fonts range from plain ASCII and Latin1 up to Latin Extended-A, which basically just covers most western characters. If you wanted to add support for Greek characters, you'd need to either regenerate all fonts to add the glyphs you need, or you'll have to create a separate bitmap for just Greek characters. That would be like reinventing codepages.

2. TrueType and FreeType

A few decades ago this problem was addressed by designing a vector-based font format, which we now know as TrueType. I'll spare you the details, but the gist of it is to use vector outlines to describe all the style variants for every glyph in the font.

There are only two issues here: 1) Getting a font that supports all the glyphs you need. 2) Reading these complex font files and turn them into an image.

First is easily solved, the web is full of TrueType fonts: http://www.fontsquirrel.com/

Second is a lot harder, but luckily enough, we have the FreeType library: http://www.freetype.org/

Still though, Freetype isn't designed to be a full-fledged text displaying engine. All it does for us is a specific glyph image and some sub-pixel data to position it correctly.

The rest is all up to us. We have to actually do the hard thing and get all this data through OpenGL 3.1 and onto the screen.

Note: Due to some specific design decision, using an existing library like freetype-gl was out of the question and as we can see later, the end result is better optimized to our needs.

3. Runtime texture atlas

This is exactly like bitmap fonts, with one major difference - we generate the texture atlas during runtime to suit our needs. If we need an emboldened font, we'll only generate that and be done with it.

The first step is to generate the basic ASCII subset, which is [0..127] inclusive, which is actually easy enough to do. We use a very basic shelf-styled packing method to create an 8-bit RED channel bitmap with just the size we need. The only requirement set by OpenGL is to align image width to 4-byte boundary.

We use a neat font called 'Anonymous Pro' - a monospace font that looks perfect for lots of text:

Using freetype library, some programming hocus-pocus and the simplest shelf-packing algorithm, we get a neat 1-channel bitmap for our 'Anonymous Pro 12px':

Can you notice the extra clarity compared to the 'Serif 12px' texture font? If you zoom in really close, you'll actually notice that the characters are neatly anti-aliased and also somewhat distorted. This is because 'auto-hinting' was enabled in order to make the small font look sharp and clear - making it a lot easier to read at smaller sizes.

4. Glyphs

We also need to store information about every character - or glyph in the atlas, otherwise we wouldn't know how to display it:

struct Glyph
{
  ushort code;  // unicode value
  ushort index; // freetype glyph index

  byte width;  // width of the glyph in pixels
  byte height; // height of the glyph in pixels
  byte advance; // number of pixels to advance on x axis
  char bearingX; // x offset of top-left corner from x axis

  char bearingY; // y offset of top-left corner from y axis
  byte textureX; // x pixel coord of the bitmap's bottom-left corner
  ushort textureY;// y pixel coord of the bitmap's bottom-left corner
};

Since we could have a lot of these glyphs (several thousand in fact), it's in our (and L2 cache) best interest to keep them small. In this case, each glyph has been optimized to be exactly 12 bytes in size and keep all the needed information to render and place a glyph.

To make glyph lookup faster, they are placed in a vector and sorted by the unicode character value. Each lookup is done with binary-search, which keeps it consistent and very fast. We should also remember that OpenGL textures are bottom-up, so pixel 0 would be the bottom-most and pixel N the topmost.

5. OpenGL 3.1 and a custom Text Shader

Now that we have everything we need, we actually have to display the glyphs on the screen. In OpenGL, everything is displayed by using Triangles and the same applies to this case. With some additional testing, I found that using a single Vertex Buffer Object with 6 vertices per glyph is the fastest and most memory efficient version. This is mostly due to the small size of the Vertex2D structure:

struct Vertex2D
{
  float x, y; // position.xy
  float u, v; // coord.uv
};

The generated glyph Quad itself looks something like this:

The simplest fragment shader to display our text in any color:

#version 140 // OpenGL 3.1
varying vec2 vCoord;			// vertex texture coordinates
uniform sampler2D diffuseTex;  // alpha mapped texture
uniform vec4 diffuseColor;		// actual color for this text

void main(void)
{
	// multiply alpha with the font texture value
	gl_FragColor = vec4(diffuseColor.rgb, texture2D(diffuseTex, vCoord).r * diffuseColor.a);
}

Which basically gives our Quad a color of diffuseColor.rgb and makes it transparent by using the font-atlas texture.

Once displayed on the screen, both filled and wireframe views of the same text:

Great! We now have a neat mono-space font perfect for displaying readable text.

6. Freetype

If we dig through some Freetype documentation, we will notice that the library also supports:

• Glyph stroke - Will give a neat outline to our fonts
  
• Glyph outline - You can just render an outline of a character.
  
• Glyph emboldening - You won't need a 'bold' font.
  
• Glyph slanting - You can generate italic text on the go.
  
• Auto-hinting - Makes small text more readable (http://en.wikipedia....ki/Font_hinting)
  

Using some simple code-sorcery, we can also add text shadows, which makes small text several times more readable on pretty much any background.

Stroke effect (48px white font + 3px black stroke):

Outline effect (48px white font + 1.5px outline):

Shadow effect (48px white font + 2px black shadow):

7. Magic of Stroke and Shadows

The stroke and shadow effect can have their own Outline color, in order to achieve this, we actually need to use another channel - Green. So for stroked and shadowed text a 2-channel Red-Green texture is generated instead of a 1-channel Red.

The best way is just to show the generated Red-Green texture:

You can see how powerful this sort glyph imaging is, since you can combine any kind of Diffuse and Outline colors to create your text effects. Best of all, fonts that don't use this effect still use a single 1-channel Red bitmap.

Even though the glyphs look really sharp and anti-aliased, they can be made blurry by just lowering the Outline alpha.

The shader used to combine this effect and work consistently for both 1-channel and 2-channel bitmaps, looks like this:

#version 140 // OpenGL 3.1
varying vec2 vCoord;		// vertex texture coordinates
uniform sampler2D diffuseTex;	// alpha mapped texture
uniform vec4 diffuseColor;	// actual color for this text
uniform vec4 outlineColor;	// outline (or shadow) color for the text

void main(void)
{
	vec2 tex = texture2D(diffuseTex, vCoord).rg;

	// color consists of the (diffuse color * main alpha) + (background color * outline alpha)
	vec3 color = (diffuseColor.rgb * tex.r) + (outlineColor.rgb * tex.g);

	// make the main alpha more pronounced, makes small text sharper
	tex.r = clamp(tex.r * 2.0, 0.0, 1.0);

	// alpha is the sum of main alpha and outline alpha
	// main alpha is main font color alpha
	// outline alpha is the stroke or shadow alpha
	float mainAlpha = tex.r * diffuseColor.a;
	float outlineAlpha = tex.g * outlineColor.a * diffuseColor.a;
	gl_FragColor = vec4(color, mainAlpha + outlineAlpha);
}

It looks a bit more complex, but the idea is simple: combine R and Diffuse, combine G and Outline, add them together.

8. Going Unicode

Now it's nice that we could achieve all this with plain ASCII, but we still haven't touched Unicode. The main reason it's so hard to support Unicode, is because there are thousands of combinations of different glyphs. It's impossible to generate all of them into a single texture - you'll easily overflow the 16384px limitation of OpenGL textures. Worst of all, even if you generate all of them, you'll probably only use 5% of them.

This is also something that freetype-gl is bad at. It's only useful for generatic a limited subset of glyphs into a single texture. If you want to support unicode, you'll go crazy with such a system.

To be honest, I also had to scratch my head quite a bit on how to solve this. If I were to use regular texture coordinates [0.0 - 1.0] on the glyph quads, they would immediatelly break if I resized the font-atlas texture. So using any conventional means pretty much flies out of the window.

The solution is pretty obvious, I have to use pixel values for the texture coordinates and generate the actual texture coordinates in the Vertex Shader. Now if I add glyphs to the end of the texture, thus resizing it, I won't break any glyph coordinates. The Vertex Shader:

#version 140 // OpenGL 3.1
attribute vec4 vertex;	// vertex position[xy] and texture coord[zw]
uniform mat4 transform;	// transformation matrix; also contains the depth information
uniform sampler2D diffuseTex;	// alpha mapped texture
varying vec2 vCoord;	// out vertex texture coord for frag

void main(void)
{
	// we only need xy here, since the projection is trusted to be
	// orthographic. Any depth information is encoded in the transformation
	// matrix itself. This helps to minimize the bandwidth.
	gl_Position = transform * vec4(vertex.xy, 0.0, 1.0);

	// since texture coordinates are in pixel values, we'll need to
	// generate usable UV's on-the-go
	// send the texture coordinates to the fragment shader:
	vCoord = vertex.zw / textureSize(diffuseTex, 0);
}

The code for generating text looks like this:

Font* font = new Font();
font->LoadFile("fonts/veronascript.ttf", 48, FONT_STROKE, 3.0f);
VertexBuffer* text = new VertexBuffer();
font->GenerateStaticText(text, L"VeronaScript");

If I were to include any unicode characters like õäöü, the font atlas would automatically generate the glyphs on-demand and add them into the atlas.

font->GenerateStaticText(text, L"Unicode õäöü");

Here's a simple before-after comparison:

With this neat feature, we can support the entire unicode range, as long as the entire range isn't in actual use. If the texture size reaches its limit of 16384px, no more glyphs are generated and all subsequent glyphs will be displayed using the 'invalid glyph'.

9. Addendum

In the first paragraph, four problems of current 0 A.D. texture fonts were raised. Now we have provided solutions to all of them:

1. Freetype allows dynamic font scaling.
   
2. Freetype provides dynamic font effects (stroke, outline, embolden, italic).
   
3. Unicode support depends on the .ttf font. The provided method ensures reasonable unicode support.
   
4. Freetype provides auto-hinting, which drastically improves small text readability.
   

The new Freetype OpenGL font system solves all these issues by introducing truly scalable vector font support, with any required font effects and full Unicode support. From my own testing I've ascertained that using shadow style drastically improves readability of text.

Regarding performance:

Even with large volumes of text, there was no noticeable performance hit. Volumes of text that are all in a single Vertex Buffer Object especially benefit from very fast render times. The more text you can send to the GPU in a single VBO, the faster the rendering should be in theory.

What's still left to do:

1. Implement embolden, slanting, underline, strikethrough.
   
2. Improve the naive shelf-packing algorithm to achieve better texture space usage. (performance)
   
3. Improve font-face loading to share between font-atlases. (performance)
   
4. Buffer all static text into a single Vertex Buffer Object, render by offset into buffer. (performance)
   
5. Implement this beast into 0 A.D.
   

I hope you liked this little introspect into the world of real-time text rendering.

Regards,

- RedFox

About Jorma Rebane aka 'RedFox': Jorma is a 22-year old Real-Time Systems programmer, studies Computer Engineering at Universitas Tartuensis and enjoys anything tech related. His weapon of choice is C++.

Edited June 6, 2013 by RedFox

[fabio]

Nice post!

But will this require OpenGL 3.1 support? 0 A.D. currently only needs 2.1 and also still works with OpenGL 1.x (though it will probably be dropped).

[RedFox]

Nice post!

But will this require OpenGL 3.1 support? 0 A.D. currently only needs 2.1 and also still works with OpenGL 1.x (though it will probably be dropped).

This is a good question actually. If I change the shaders to target #version 120, it will work perfectly fine for OpenGL 2.1 and OpenGL ES 2.1. There's actually no reason it can't target those versions. The only reason I stated OpenGL 3.1 is because it completely relies on modern OpenGL, not on the old deprecated immediate mode.

It can't target older OpenGL versions though, since I'm using NPOT (non power-of-two) textures.

Regarding OpenGL versions in general, I think 0AD should target GLES 3.0+ and OpenGL 3.0+, since older hardware won't be able to run the game anyways.

Edited June 6, 2013 by RedFox

[Yggdrasil]

Nice post, but i have some concerns: Your unicode support is very limited when it comes to complex scripts [1][2]. I don't think texture-mapping of glyphs will work here. There are just too many combinations.

That's why Ykkrosh proposed to use pango which uses harfbuzz internally to support complex rendering [3]. It also uses fontconfig to search for glyphs in multiple fonts. So you don't need to know in which font a specific glyph is stored.

With pango you render (through cairo) your whole text snippet into a texture which you could cache for later usage. Of course this is slower than texture-mapping but i doubt it'll be a bottle neck.

[1] https://en.wikipedia.org/wiki/Complex_text_layout

[2] http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=CmplxRndExamples&_sc=1

[3] http://www.wildfiregames.com/forum/index.php?showtopic=13790&st=40#entry236362

[RedFox]

Nice post, but i have some concerns: Your unicode support is very limited when it comes to complex scripts [1][2]. I don't think texture-mapping of glyphs will work here. There are just too many combinations.

You have a fair point in Complex Text Layout limitations. Right now only common glyph based text could be displayed and that's bad for most eastern fonts. However with the current layout engine, I could easily create render-to-texture text instead of glyph sequences (I'm already doing it when creating the font-atlas texture).

Taking a look at the Pango documentation, it seems like a very complex library and will definitely be a pain to use.

However, we can use Harfbuzz separately to convert UTF-8 text into meaningful Unicode values and then generate text during runtime. We can easily toggle between font-atlas rendering for latin, greek, cyrillic and use render-to-texture text for arabic, indian, chinese...

There's even a cool example for this here: https://github.com/lxnt/ex-sdl-freetype-harfbuzz/

Right now I'd rather keep Complex Text Layout at the end of the priority list (it's just very fancy text rendering), but if it's deemed to be a 'must-be' feature by Michael, we could integrate Harfbuzz.

[Mythos_Ruler]

IMHO, making sure Arabic and Chinese all render correctly is not a top priority. IIRC, Age of Kings was big in Korea at one point even without Korean script support. As long as the code can be extended in the future to include these languages, then we're good (possibly for Part 2 after Part 1's release has gone wide and is a proven success?). For now, focus on Western fonts and languages. That's what I think anyway.

[Yves]

Very interesting and well written article!

[RedFox]

Today's progress report on the custom layout engine:

-) Improved font-face loading:

Before I had to re-load a TrueType font file every time I needed a specific style and size font. Now it's optimized to work as it should - by using a single TrueType face and processing it through specified sizes.

Of course, it required quite a lot of structural change again, but in the end, the

'API'

is a lot easier to use:

FontFace* face = new FontFace("fonts/veronascript.ttf");
Font* font = face->CreateFont(48, FONT_STROKE, 3.0f, 96); // fontHeight, style, outlineParam, dpi
Text* text = font->CreateText(L"Hello text!");

// ... render text

-) DPI awareness:

With a multitude of different DPI settings out there, it was logical to make the font system DPI-aware. In this case, if the user has changed the default system DPI value, the fonts will be rendered accordingly. Increasing the DPI values will make the text look bigger

(and thus more readable)

.

I should emphasize again, that the DPI setting itself is a system-wide property and is

changed by the user

to make the system text bigger. The default DPI value in windows is 96.

Here's a sample with DPI 108:

-) What's still left to do:

• Implement embolden, slanting, underline, strikethrough.
  
• Improve the naive shelf-packing algorithm to achieve better texture space usage. (performance)
  
• Buffer all static text into a single Vertex Buffer Object, render by offset into buffer. (performance)
  
• Implement this beast into 0 A.D.
  

[serveurix]

IMHO, making sure Arabic and Chinese all render correctly is not a top priority. IIRC, Age of Kings was big in Korea at one point even without Korean script support. As long as the code can be extended in the future to include these languages, then we're good (possibly for Part 2 after Part 1's release has gone wide and is a proven success?). For now, focus on Western fonts and languages. That's what I think anyway.

What about mods with vernacular names ? I think there are already asian history mods of 0AD on the way.

[graubart]

Any progress on this?

[historic_bruno]

Any progress on this?

Most of the progress/discussion occurred on IRC, such as here. Basically, rewriting the font system would have inevitably resulted in a lot of bugs and limitations but not a lot of new features. It's kind of weird to go through all that trouble and just omit complex text layout, for example, especially when the existing font engine already has Unicode support and supports as many effects as we can think of by modifying Python scripts rather than C++. All that's needed for prettier fonts (short of a redesign adding far more than the above approach) is for artists to request new fonts or effects somewhere like Trac, and a programmer to generate the textures or walk them through the steps for doing so. I haven't seen that in the last 3 years, so if someone is unhappy with our fonts, they aren't documenting it well, if at all.

[GunChleoc]

IMHO, making sure Arabic and Chinese all render correctly is not a top priority. IIRC, Age of Kings was big in Korea at one point even without Korean script support. As long as the code can be extended in the future to include these languages, then we're good (possibly for Part 2 after Part 1's release has gone wide and is a proven success?). For now, focus on Western fonts and languages. That's what I think anyway.

We already have an Arabic and a Chinese translator signed up, so I would think this should be ready for gold, so their work can be used.

[Ykkrosh]

All that's needed for prettier fonts (short of a redesign adding far more than the above approach) is for artists to request new fonts or effects somewhere like Trac, and a programmer to generate the textures or walk them through the steps for doing so. I haven't seen that in the last 3 years, so if someone is unhappy with our fonts, they aren't documenting it well, if at all.

I think Mythos has moaned about it a bit, though partly that was because he'd force-enabled FXAA which antialiased all the text and made it hideous (as it does in other games too) . There are almost certainly things that would be useful and straightforward to improve, though. I did some experiments recently like this, with different fonts and with shadowing instead of stroking and with the different hinting options - it's easy to do that to see what things will look like, and to regenerate the game's fonts, but someone needs to make an informed judgement call on what would be best.

(It's fairly easy to add support for non-bidi non-complex-shaped non-CJK scripts (Cyrillic is probably the main one) in the current system too - they get packed reasonably efficiently (so we don't really need to generate the textures at runtime), and it just needs a few improvements to some Python scripts. But bidi (Arabic, Hebrew) is non-trivial since it'll have to interact with our GUI engine's text layout code, CJK would possibly make the textures too large and I think it needs language-sensitive font selection (not a straight codepoint->glyph mapping), and complex shaping (prettier Arabic, Devanagari, maybe nicer English with sub-pixel hinting) is impossible with a texture-full-of-glyphs approach and would need a totally different design. So I think it's worthwhile doing the easy things now, and it would be nice to eventually do a totally different design that fixes all those problems, but it's not worth doing a new implementation of a glyph-texture design since it won't get us any closer to fixing those problems.)

[historic_bruno]

I think Mythos has moaned about it a bit, though partly that was because he'd force-enabled FXAA which antialiased all the text and made it hideous (as it does in other games too) . There are almost certainly things that would be useful and straightforward to improve, though. I did some experiments recently like this, with different fonts and with shadowing instead of stroking and with the different hinting options - it's easy to do that to see what things will look like, and to regenerate the game's fonts, but someone needs to make an informed judgement call on what would be best.

Right, I just wish there were Trac tickets for that, to clarify what is needed and what work has been done so far Otherwise I don't expect any progress on fonts, there are plenty of other things that need doing. Did you have to modify fontbuilder to add new effects/settings, if so can you make a git branch or attach a patch somewhere?