Palaxin's Heightmap Guide

[Palaxin]

Introduction

Spoiler

If you do not already know the heightmap import feature of the Atlas map editor, please read the official wiki page first before continuing here. It is especially stated there that “[c]urrently the heightmap importer doesn’t have any configuration, so if import doesn't produce the desirable result, the source image should be edited in an image editor and re-imported.”

This guide will give you a helping hand with the required image editing. In the Beginner section you will find the basics everyone needs for heightmap manipulation with easy step-to-step instructions and many illustrations. The Advanced section includes topics not everyone may use, but the necessary steps are still not too complicated for anyone who doesn't like coding. In the Expert section I will even show how to use direct hacks to the PMP file which stores the map height data. However it doesn't contain strict rules and you need to be able to write and run scripts.

I hope you enjoy this guide and strongly appreciate any constructive criticism, proposals, discussions and comments especially since this is my first guide At this point I would like to thank @FeXoR, @Garaf and @niektb.

Basics

Spoiler

Software

This guide comes without any liability regarding the used software. It might damage or cause instabilities to your system. My operating system is Windows 10 Pro and I will give instructions only for the programs listed below.

The minimum requirements for heightmap manipulation described in the Beginner and Advanced sections are

• an installation of 0 A.D.
• a web browser
• an image editing tool, preferably GIMP

Heightmap manipulation techniques in the Expert section additionally require

• a text editor, preferably Notepad++
• the text and hex editor PSPad
• the hex editor HxD
• a working environment for running scripts and some knowledge for writing them

Heightmap types

The basis for all further processes are the original grayscale heightmaps, so they should be chosen carefully. There are three different types:

Topographic images represent the elevation of the land surface. They usually use black for ocean level and lighter gray up to white for mountains. Using topographic images alone is only useful when in your area of interest there are no to very little water areas as you will get very shallow waters.

Bathymetric images represent the elevation of the ocean floor. They usually use white for all land areas and darker gray down to black for deep trenches.

Combined topographic and bathymetric images both depict ocean floor usually with darker gray and land surface with lighter gray. Quality images of this type are very rare so most of the time you will need to combine topography and bathymetry manually. For more information have a look at the Advanced section.

In the further mapmaking process you may also want to use photographs from airplanes or satellites as a reference for terrain texturing. However the globe has changed a lot in the last 2,000 years so it is strongly advised to take historic information into account as well.

Heightmap sources

If you are looking for large scale real world data, the NASA Blue Marble image series is an excellent source. It includes topography and bathymetry heightmaps as well as satellite based composite photographs of the earth. Examples are shown above.

For topography and bathymetry you can choose between two resolutions and image formats. The 21,600*10,800 pixels PNG images have a resolution of 1/60° = 1 arc minute which means one elevation data point every 1,855 m or 6,087 ft at the equator. For more details there are 8 GeoTIFF images available with 10,800*10,800 pixels each which also cover the whole globe when assembled together in the following way:

A1 | B1 | C1 | D1
A2 | B2 | C2 | D2

Their resolution is 1/120° = 1/2 arc minute = 30 arc seconds which corresponds to 928 m or 3,044 ft at the equator. These global maps contain very much details. Even a 32*32 pixels (29.7 km or 18.4 mi) piece suits for the Tiny map size, 64*64 pixels (59.4 km or 36.9 mi) for Medium and 128*128 pixels (119 km or 73.8 mi) for Giant. For smooth terrain you can even halve that sizes. You just have to scale the piece by a factor of 4 or 8, respectively. If your image program doesn't already interpolate between the pixels, you need to add a blurring filter.

Due to image file size you should right click the link and directly save the picture as it may cause instabilities to your browser. For bathymetry, don't use the colored versions.

If you are interested in heightmaps for even smaller areas, you will find detailed data in this digital elevation data search engine. I can really recommend the 15 arc second resolution (464 m or 1,522 ft at the equator) and the 3 arc second resolution (92.8 m or 304 ft at the equator) topographic data collected by Jonathan de Ferranti. You will need to convert the data to a suitable image format, e.g. with GDAL. Have a look at Garaf's posts for a bit more information.

If you do not already have samples, you can use the ones below. Alternatively you can paint your own grayscale heightmaps in GIMP.

    

Beginner

Spoiler

Selection and scale

First you need to crop your image to the desired area and scale it to the desired size. In the GIMP menu bar, click on Windows → Toolbox and select the elliptic symbol Ellipse Select Tool.

After that, go to the menu bar again and open the Tool Options dialog via Windows → Dockable Dialogs → Tool Options. Check the Fixed box and select Aspect ratio in the field aside. The ratio must be 1:1. Make sure that Mode is set to Replace the current selection (the single square filled with red).

    

Now drag your cursor over the image while holding Left Click. You can estimate the area a circle map and a square map would include. Play around until you are satisfied. Be aware that the areas next to the map border won't be accessible for units and buildings in the game. If you want to combine topography and bathymetry files as described in the Advanced section, additionally note the pixel values for Position and Size, so you can repeat the exact selection in the other image.

    

Select Image → Crop to Selection and then Image → Scale Image. Enter 128, 256 or 512 pixels for both Width and Height, depending on the map size you want. Finally select Filters → Blur → Gaussian Blur with a horizontal and vertical blur radius of about 4-8 pixels. A higher amount of pixels means your map surface will be smoother which allows better building placement.

    

Image → Mode → Grayscale should be marked, if it's not the case, do it now. Finally save your file with File → Export As and choose an appropriate name ending with .png so the image can be imported in Atlas.

Height range

When importing a heightmap with white and black areas, you probably will find that using the whole height range will lead to very steep slopes, giant mountains and/or giant sinkholes. Therefore you need to change the grayscale range in your image.

Open the heightmap in GIMP and make sure that Image → Mode → Grayscale is marked. Select Colors → Brightness-Contrast and click on Edit these Settings as Levels in the opening dialog. In order to see more details, expand the window to the majority of your screen. Under Input Levels you can see the distribution of different shades of gray in the image. Higher columns mean the respective shade occurs more often in the image. Now play around with the Output Levels slide control. You can change the minimum and maximum gray value and thus the minimum and maximum height in your final map. I recommend to set the minimum value to 0, because the skybox (often seen on screenshots with flat camera angles) is only present for low height ranges. Set the maximum value to about 64 (fourth of the full scale) or less.

Save your edited file clicking File → Export As with a new name ending with .png. Import the heightmap image in Atlas to check the result and re-edit it in GIMP until you are satisfied. For each new version, always start with the original image to minimize the loss of information.

Advanced

Spoiler

Curved surface corrections

There is an issue with the proportions on large scale heightmaps like the global maps from the NASA Blue Marble series. It can be neglected for maps depicting areas smaller than roughly about 1,000*1,000 miles or 1,600*1,600 km. The problem with earth's surface is that it can't be mapped on a flat surface without changing scale and/or proportions. There is no perfect solution, but if you are taking only a part of such a big map, there is a great potential for enhancement. Near the equator or the vertical center of the original map we don't need to do corrections, but the closer your area of interest is to the upper or lower edge, the greater the deviations of size and/or proportion are in that area.

Now, take the center of your area of interest and determine its deviation from the center of the original map in degrees. You can use a globe or Google Maps for this task. Calculate 1/cos(degrees) and note that number. In GIMP, click on Image → Scale Image and then on the chain right of the Width and Height field. Then multiply your calculated number with the height and enter the new height. The proportions in your map center should be correct again, and they will deviate more and more to the upper and lower map edge (but much less than before).

Of course you need to do this scaling before you crop your map to the final square piece, or you need to crop it again.

Merging topography and bathymetry

Often it will be necessary to combine bathymetric and topographic heightmap images. If they have the same size and depict exactly the same area, this task is quite simple. Open the bathymetry file in GIMP and copy the topography image on top of it. Open the Windows → Dockable Dialogs → Layers dialog and set Opacity to 50.0. Now you have combined the two heightmaps with the cost of halving the precision. Before their fusion they had 256 steps of gray each, now only 256 steps in total. I will show how to eliminate this information loss in the Expert section.

Now it can get pretty complicated if your heightmaps don't fit 1:1 on top of each other. You will need to scale one to the other. You can get the best estimation for the right scale by examining steep shorelines. You may also edit the images to make the shoreline more visible. For example you can use the upper slide control in Colors → Brightness-Contrast → Edit these Settings as Levels to accomplish this.

Expert

Spoiler

Here I will show you how to extract height information directly from BMP bitmap images and write them to the PMP file which stores the height data of the actual 0 A.D. map.

This is only one possible solution, and it is not the most efficient nor the most elegant. It's just working with the tools and skills available to me. I would be glad if some developer could use this as an inspiration for a better solution.

Motivation

Atlas imports 8-bit grayscale heightmaps which are encoding 256 possible heights. These heights are mapped on the whole height range supported by 0 A.D. There are 65,535 possible heights corresponding to about 712 m meaning one height step corresponds to about 1.1 cm. However, the grayscale heights produce steps of about 2.78 m. If you have a close look at imported height maps this leads to a terrace like style especially at smaller slopes. Making direct changes to the PMP file in theory allows access to the full range of 65,535 possible heights. However, you usually find 8-bit and not 16-bit grayscale heightmaps so the precision increase is not that big. In most cases it is still big enough to make the terrace steps so small that they are not noticed in-game.

A typical map in 0 A.D. only uses less than an eighth of the possible height range (because this was the old height limit, see ticket #3112), newer maps may also use about a quarter of the range. This means there are fewer than 32 discrete height values (up to 64 for newer mountainous maps) when using the standard heightmap import. If you combine topographic and bathymetric height data and write it to the PMP file, you can have a total of up to 2*256 = 512 discrete heights within the same range. This means a precision increase of about 5-15 depending on the number of shades of gray and in-game height range. The height steps shrink from 2.78 m to about 19 to 56 cm in size.

Currently you need to do a lot of extra work for a quality map after the heightmap import due to the limited precision. But if you use the methods described in the next sections, you can reduce that work to a minimum.

Extracting height data from BMP images

Usually images are stored in a binary file format so you can't simply read them with a text editor. However, uncompressed data can be accessed easily with a hex editor. I recommend to export the heightmap as grayscale BMP file in GIMP and open it with PSPad. Click on View → Hex Edit MODE if you are still in the text edit mode. Wikipedia describes how to interpret the shown data. It is important to know that the data is stored in little-endian format meaning the least significant byte is found most left. Some important bytes in the BMP file format are:

• 11-14: 32-bit unsigned integer, offset or starting address of the actual bitmap image data
• 19-22: 32-bit integer, bitmap width in pixels
• 23-26: 32-bit integer, bitmap height in pixels

If the last number is positive, we have a bottom-up bitmap where the data starts in the lowest line and ends in the highest. This is important for our use since the PMP file also stores the data bottom-up. Otherwise we would have a top-down bitmap.

We would expect that the data block has a size of the pixel width*height since GIMP exports to 8-bit grayscale. However, if the width is no multiple of 4, the rest is filled up with zeros. This is important because the PMP map size is (mapsize*16 + 1), so there are three empty bytes following the end of each pixel line. You need a 129*129 pixels image for the Tiny map size, 257*257 pixels for Medium and 513*513 pixels for Giant.

Script calculations

With the right scripting languages you can directly readout data from binary files. However I'm a JScript noob. No it's not JavaScript, but a scripting language invented by Microsoft. I didn't choose it, but had to learn it for my Bachelor thesis and it's the only scripting language I'm comfortable with at the moment. The bad thing about JScript is that it only supports the readout of text files. If you need a text file like me, copy the hexadecimal code into Notepad++ and save it in text format. This procedure is done for both the topography and the bathymetry file.

With the knowledge about BMP files from the previous section, the script extracts the 8-bit height values of topography and bathymetry, chooses one of them and converts them to 16-bit values considering some boundary conditions. These include the minimum and maximum height of the output and the number of additional steps inserted between land and ocean. This is an useful feature for maps with very shallow waters as units won't just walk through the ocean. The calculated data is then written to a new text file. You can have a look at my script bmp-to-pmp-conversion.js.

Writing height data to PMP files

Our wiki contains all information you need to know about the PMP file format which stores the height and terrain texture information of the 0 A.D. maps.

The easiest way to get an empty PMP file for your needs is to open the Atlas editor, generate a new random map Wall Demo of the desired map size, remove all structures and lift all the terrain to the maximum height. This way the data section will be clearer visible when you open the file with HxD. This hex editor supports the selection of huge file parts with Shift + Left Click. However, unlike PSPad, HxD inserts whitespaces when you copy hex code to a text file, so I'm using both editors.

Now copy the processed data of the script with Notepad++ (or with a hex editor if you didn't use text files) and paste it in the right place in the PMP file. You should skip the first 16 bytes and select all data until the F-sequence stops before pasting the new data. If no This operation changes the file size message is shown you probably haven't done much wrong yet. Save your edited file and open the map again in Atlas and voilà – there it is!

Changelog

Spoiler

version 0.5 - 06/26/2016

• Introduction: also thank FeXoR and niektb, remove outdated picture announcement
• Basics - Heightmap sources - 2^nd and following paragraphs:
  add information about GeoTIFF images and resolution in degrees, arc minutes/seconds, (kilo)meters and miles/feet
  link/recommend high resolution data collected by Jonathan de Ferranti
• Advanced - Curved surface corrections - 2^nd paragraph: link Google Maps
• Expert - Motivation - 2^nd paragraph: add information about old height limit and correct some numbers
• Changelog: added as a new spoiler
• wording and formatting improvements

version 0.4 - 06/06/2016

• Introduction: thank Garaf
• Basics - Heightmap sources: link DEM search engine, GDAL and Garaf's posts about its usage

version 0.3 - 06/05/2016

• Pro: rename to Expert

version 0.2 - 05/28/2016

• Basics: insert pictures, simplify instructions by removing unnecessary steps

version 0.1 - 05/28/2016

• initial commit

 

Edited June 25, 2016 by Palaxin

[FeXoR]

@Palaxin Very nice!

I added a link in the Heightmap Import section of the Atlas Manual.

If you want you can extend that section yourself if you feel some informations are missing there

EDIT: I wonder though if anyone could make a living with that "Pro" hints ... for that's what "professional" means AFAIK ^^

 

[Palaxin]

Added illustrations to the Basics and Beginner sections

4 hours ago, FeXoR said:

EDIT: I wonder though if anyone could make a living with that "Pro" hints ... for that's what "professional" means AFAIK ^^

You suggest to rename it to something else? E.g. Advanced II? Or a more content related caption like .pmp file hacks or scripting or something similar? BTW thanks for linking the guide

Edited May 28, 2016 by Palaxin

[niektb]

Expert could work too instead of Pro (though I've no problems with Pro)

[FeXoR]

I'd go with "Advanced with Scripting" I guess

Edited May 29, 2016 by feneur
Scriting -> Scripting

[Palaxin]

I finally went for "Expert", thx for the feedback

[Scallact]

Hi everyone,

Thanks to Palaxin for his tutorial. I was frustrated by the quality of the Atlas import, and have decided decided to tackle the expert method. The problem for me was that a script is needed, because of the way the .bmp files are encoded.

After some research, I found out that .tif file have a much friendlier encoding for our use, and we can literally copy-paste the pixels block inside the .pmp as is. As long as the tif file is correctly formatted. The good news is that GIMP nowadays supports 16 bits integer images and export!

I wrote a detailed tutorial, please tell me if something is unclear. Here we go:

 

Preparation

Create and save a basic map with the Atlas. Resize it and note the size, it's very important.
The map is made of two files: a .pmp and a .xml. In this tutorial, we will paste the height map into the pmp.

Sizes

Your initial height map can be any size. I've included a resize step in the process. But, depending on the size selected for your map in the Atlas, the final .tif height map must have the right number of pixels.
Here are the corresponding sizes in pixels:

Tiny:      129x129
Small:     193x193
Normal:    257x257
Medium:    321x321
Large:     385x385
Very Large:449x449
Giant:     513x513

Note 1: It is essential that all your exports are: uncompressed tif images, grayscale, with 16 bit integer precision, and no alpha channel. Every other setup changes the number of digits for each pixel, and will mess up your map quite badly.

Note 2: With the method here, the map will become a mirror copy of your original height map. If you don't want that, you can add the following point to the "Export operations":
2a) Image > Transform > Vertical mirror

I. Elevation file export with GIMP

IA) (only once per size) Create a a uniformly white image and follow the export operations. This is the .tif file used to find the extent of the pixels. You need to do that only once for each map size. I suggest you name it "white_template_size.tif".

IB) Open your elevation file in GIMP and follow the export operations. If it's an .xcf file, be careful NOT to hit Save at any point during the operations.

Export operations

1) Image > Flatten Image
        this ensures there is no alpha channel and only one layer

2) if you original file is not the right size:
        Image > Scale Image, select the "pixel" unit and enter the right size.

3) Image > Mode > Grayscale

4) Image > Precision
        choose 16-bit integer, linear light

5) (IB only) File > Export
        choose a name for your file, add something like "_full_height", and add the .tif extension
        choose Compression : none, and uncheck all boxes
        export
        note: you will probably have to try different curve settings, so it's useful to keep and reopen the "full_height" file for subsequent tries.

6) (IB only) Colors > Curves
        check the "linear" icon on the top right
        select the top-right point of the curve
        in the "output" field, enter a small number (I usually start with 5.00) and validate
        note: you can use Colors > Levels as well. Select "linear", enter your value in the rightmost "output levels" field.

7) File > Export As...
        choose a name for your file and add the .tif extension (let's call it MyMap_size.tif)
        choose "Compression : none", and uncheck all boxes
        export

II. Height map import with hexadecimal editor
 

You'll need an hexadecimal editor which allows to select large blocks with SHIFT. On Linux, I use the Bless Hex Editor.

1) Open "white_template_size.tif" and MyMap_size.tif with the hex editor side by side.
        note: on the editor, the cursor's position is referenced as "Offset" followed by an hexadecimal address

2) On the white_template file, notice that most of it is filled with "FF". This tells us where the actual pixels start and end. Go to the end of the file and place the cursor just after the last "FF". It's usually a "12". Read the offset.

3) On the MyMap file, go to the end of the file, and place the cursor at the exact same offset. It should also be a "12".

4) Scroll back to the start of this file. If you observe the white_template, you should notice that there are 8 positions before the "FF" start. Back to MyMap, count 8 double-digits and SHIFT+clic on the next one. A big block should be selected. COPY.

5) Open the .pmp file with the hex editor. Place the cursor at offset "10". It means 16 in base 10, so the cursor is actually at the 17th position.

6) Press the "INSERT" key. The info bar should now indicate "OVR" instead of "INS".

7) PASTE.

8) Save, open your map in Atlas. The relief is probably too accentuated, or too flat. Repeat the operations, starting from Ib) 6) above. Adjust the output of the curve with a different value. Don't forget to always set the curves in linear mode.

 

Cheers,
Pascal

 

Edited September 20, 2021 by Scallact

[Scallact]

When I have some time, I'd also like to do a guide on how to design very nice height maps with GIMP. Here's an example enhanced with some shading for better viewing.

P.S: uploading a 33kb jpg fails, I'll try another time.

[wowgetoffyourcellphone]

My frustration comes from the exaggerated affect applied to the map in Atlas. This happened when the number of height units was increased some years ago and was never fixed. As a result, you have to use a very small range within the grayscale image.

[Scallact]

Just now, wowgetoffyourcellphone said:

My frustration comes from the exaggerated affect applied to the map in Atlas. This happened when the number of height units was increased some years ago and was never fixed. As a result, you have to use a very small range within the grayscale image.

I take this into account. Also, with this method, we don't lose any height precision. We have access to the full 65536 discrete height values.
I strongly suggest to allway keep a copy of the original file you're working on, and only apply the linear curves in the last steps, as indicated in this tutorial.
Just don't be afraid to apply some curves with some crazy small output value. I can guarantee the 16bit exported file is still very precise! :-)

[Scallact]

Hi,
Here is an example of import with the method above.
The first image is made from the height map in GIMP with some shading and altitude gradient.
The 2 others are from the Atlas, with some basic terrain covering so that we see the relief well. The map size is medium

Cheers!

P.S: the height map is shown before resizing to 321x321px, it's much higher resolution than the Atlas shots.

 

Edited September 20, 2021 by Scallact

[andy5995]

Not wanting to go too far off-topic here, but you can also get heightmaps from existing skirmish maps for use in generating random maps. For example, when Jebel Barkel is generated, it imports a pmp file:

g_Map.log("Loading hill heightmap");
createArea(
    new MapBoundsPlacer(),
    new HeightmapPainter(
        translateHeightmap(
            new Vector2D(-12, scaleByMapSize(-12, -25)),
            undefined,
            convertHeightmap1Dto2D(Engine.LoadMapTerrain("maps/random/jebel_barkal.pmp").height)),
        minHeightSource,
        maxHeightSource));

One application for this method is that if you have a 2v2 skirmish map for example, one might create a random map that allows more flexibility with the number of players, using the same terrain as a nice skirmish map. There's some extra info in trac ticket 4816.

[Palaxin]

On 19/09/2021 at 9:45 PM, Scallact said:

Thanks to Palaxin for his tutorial.

Glad to hear that the tutorial is still helpful after more than 5 years And thanks for the additions, @Scallact!
Since then, there have been quite some changes and new features to map generation, as @andy5995 mentionend...

[Scallact]

15 hours ago, andy5995 said:

Not wanting to go too far off-topic here, but you can also get heightmaps from existing skirmish maps for use in generating random maps. For example, when Jebel Barkel is generated, it imports a pmp file:

[...]

One application for this method is that if you have a 2v2 skirmish map for example, one might create a random map that allows more flexibility with the number of players, using the same terrain as a nice skirmish map. There's some extra info in trac ticket 4816.

If I understand well, this is useful to transfer an existing heightmap to another (especially random maps) ?

[Scallact]

12 hours ago, Palaxin said:

Glad to hear that the tutorial is still helpful after more than 5 years And thanks for the additions, @Scallact!
Since then, there have been quite some changes and new features to map generation, as @andy5995 mentionend...

Very helpful indeed! The Atlas import function is far from perfect, and I really wanted to be able to "sculpt" my own heightmaps in GIMP and import them without loosing any quality. My initial idea was to experiment with symmetrical heightmaps, and maps where mountains/hills are not used as obstacles, but as tactical features.

I have found a more straightforward way to to the second part: heightmap import. With a small command line utility able to copy binary code from one file to the other. I will complete my post when I have time.

Of course, creating handmade heightmaps might seem a bit daunting. However, I'm more of a graphic guy, and that was the easy part for me ;-) I have developed some nice and accessible methods to build them with GIMP, and will share these in another post in the coming weeks.

[andy5995]

On 25/10/2021 at 5:28 AM, Scallact said:

If I understand well, this is useful to transfer an existing heightmap to another (especially random maps) ?

@ScallactThis is a random map I started to make. It imports the pmp file from the Corinthian Isthmus 2v2 skirmish map:

 

 

[Scallact]

1 hour ago, andy5995 said:

@ScallactThis is a random map I started to make. It imports the pmp file from the Corinthian Isthmus 2v2 skirmish map:

 

Thanks, that's what I thought it was for. Nice job!

[andy5995]

Tangram Heightmapper seems like a great source to get a heightmap. I added the link to the wiki.

[Scallact]

16 hours ago, andy5995 said:

Tangram Heightmapper seems like a great source to get a heightmap. I added the link to the wiki.

Thanks!

Complementarily, I'm currently working on a GIMP script to create customized height maps. I've made many experiments since a few months, and It's starting to show nice results. I should be able to share it in the future.

[Scallact]

Here's a preview of the GIMP plugin I'm working on.