Hacking Atlas Heightmap Import

[Palaxin]

Thanks @wraitii,

I already implemented .pmp file writing in another scripting language and just need to "copy" it to Python (version 3.5)  sanderd17's script already helped me a lot. Perhaps we could make a shared mini library "heightmap_io" (or similar) for .pmp reading and writing? Also grayscale bitmap (AFAIK .bmp is the simplest one) import/export could be included as well as .hgt reading.

Edited June 6, 2016 by Palaxin

[wraitii]

Part of the problem is that our map are 16-bit grayscale, which is a bit of an odd format. Using Pillow on Python I used a 32-bit floating point texture, saved in tiff.

 

FYI here's the bit of my script (taken as is), with the useless stuff removed:

with open(os.path.join(sys.argv[1],i.replace(".xml",".pmp")), "rb") as f:
		f.read(4) # 4 bytes PSMP to start the file
		f.read(4) # 4 bytes to encode the version of the file format		
		f.read(4) # 4 bytes for the file size

		patchSize = int.from_bytes(f.read(4), byteorder='little')

		heightmap = [[0 for x in range(patchSize*16+1)] for y in range(patchSize*16+1)] 
		textureTile = [[0 for x in range(patchSize*16)] for y in range(patchSize*16)] 

		# Height,qp is 16 bit integer divided by 92 to yield the real height
		for x in range(0, patchSize*16+1):
			for z in range(0, patchSize*16+1):
				heightmap[x][z] = int.from_bytes(f.read(2), byteorder='little') / 92

		numTex = int.from_bytes(f.read(4), byteorder='little')
		#print(str(numTex) + " textures used.")

		texNames = []
		for texI in range(numTex):
			# first 4 bytes are texture name length then read the texture name itself
			name = f.read(int.from_bytes(f.read(4), byteorder='little'))
			texNames.append(str(name,"utf-8"))

		#im = Image.new("L", (patchSize*16,patchSize*16) )
		#pixels = im.load()

		for z in range(0, patchSize):
			for x in range(0, patchSize):
				for pz in range(16):
					for px in range(16):
						texA = int.from_bytes(f.read(2), byteorder='little')

						#pixels[x*16+px, z*16+pz] = (texA)
						f.read(6) #second texture (2) + priority (4), don't care

		#im.save(sys.argv[1] + "." + i + "output.png","png")
		
		#im = Image.new("F", (mapSize+1,mapSize+1) )
		#pixels = im.load()
		#for x in range(0, math.floor(math.sqrt(mapSize))*16+1):
		#	for z in range(0, math.floor(math.sqrt(mapSize))*16+1):
		#		pixels[x,z] = heightmap[x][z]
		#im.save(sys.argv[1] + "." + i + "output.tiff","tiff")

		#EOF

 

[Palaxin]

.pmp file writing is working well with the module below. I started learning Python 2 days ago, so don't wonder if the code doesn't follow any conventions.

It is not really possible to reverse the write function as there can be only one return value and at least height AND textures are interesting. Perhaps define a map object and return one? @wraitii I also notice you don't use the struct module, is there a reason (performance, ...)? Thx for your script

import os
import struct

PMPVERSION = 6 # .pmp file version
MINTILES = 128 # tiles per line and column in the smallest valid map size
MAXTILES = 512 # tiles per line and column in the biggest valid map size
STEPSIZE = 64 # difference in tiles per line and column between two map sizes


def validMapSize(mode, array):
	"Returns True if a 2D array of integers represents a valid map size. \
	'mode' can be 'h' for heights or 't' for textures."
	
	# 16*n texture tiles, but 16*n + 1 heights per line and column
	rest = 1 if mode == "h" else 0
	
	if len(array) % STEPSIZE != rest or len(array) < MINTILES or len(array) > MAXTILES:
		return False
	for i in range(len(array)):
		if len(array[i]) % STEPSIZE != rest or len(array[i]) < MINTILES or len(array[i]) > MAXTILES or len(array[i]) != len(array):
			return False
	return True


def writeToPMP(filename, heights, textures, texturenames, texturepriorities):
	"Writes a new .pmp file and returns True if successful. \
	'heights' and 'textures' must be 2D arrays of 16-bit integers, \
	'texturenames' a string array and 'texturepriorities' an array of 32-bit integers."
	
	if not validMapSize("h", heights) or not validMapSize("t", textures):
		print ("invalid map size")
		return False
	#if not os.path.isfile(filename) or filename[-3:] != "pmp":
	#	print ("\"" + filename + "\" invalid path or filename")
	#	return False	
	
	numtiles = len(textures)
	numpatches = round ( numtiles / 16 )
	numheights = numtiles + 1
	
	# header bytes + height data + u32 array length
	filesize = 16 + 2 * numheights**2 + 4
	# u32 string length + string
	for i in range(len(texturenames)):
		filesize += 4 + len(texturenames[i])
	# texture data
	filesize += 8 * numtiles**2
	
	with open (filename, "wb") as f:
		
		# write file header
		f.write(struct.pack("4s", b"PSMP"))
		f.write(struct.pack("<I", PMPVERSION))
		f.write(struct.pack("<I", filesize - 12))
		f.write(struct.pack("<I", numpatches))
		
		#write height data
		for vHeight in range(numheights):
			for hHeight in range(numheights):
				f.write(struct.pack("<H", heights[hHeight][vHeight]))
		
		#write texture names
		f.write(struct.pack("<I", len(texturenames)))
		for i in range(len(texturenames)):
			f.write(struct.pack("<I", len(texturenames[i])))
			for j in range(len(texturenames[i])):
				f.write(struct.pack("c", texturenames[i][j].encode("ascii","ignore")))
		
		#write texture data
		for vPatch in range(numpatches):
			for hPatch in range(numpatches):
				for vTile in range(16):
					for hTile in range(16):
						# index of texturenames array
						f.write(struct.pack("<H", textures[hPatch * 16 + hTile][vPatch * 16 + vTile]))
						# second texture, not used in 0 A.D., so 0xFFFF
						f.write(struct.pack("<H", 65535))
						f.write(struct.pack("<I", texturepriorities[hPatch * 16 + hTile][vPatch * 16 + vTile]))
		f.close()
	
	return True


def writeHeightToPMP(filename, heights):
	"Writes a new .pmp file from the 2D 16-bit integer array 'heights' and a default texture. \
	Returns True if it was successful."
	
	l = len(heights) - 1
	textures = [[0 for i in range(l)] for j in range(l)]
	texturepriorities = [[0 for i in range(l)] for j in range(l)]
	
	return writeToPMP(filename, heights, textures, ["medit_shrubs"], texturepriorities)

 

[wraitii]

python 3 recently added the int8frombytes and I find it more readable. No other reasons.

Also not an expert in python, far from it.

I don't understand your question overall.

[Palaxin]

17 minutes ago, wraitii said:

I don't understand your question overall.

Sorry, it's a bit unclear indeed. I mean a readFromPMP() function (which I called the "reverse" of the write function) of course can only return one value and not "heights", "textures" etc. (the arguments of the writeToPMP() function). So if I want to extract different data from a file by using this readFromPMP() function, I will have to return an object which contains various data like height and texture arrays.

Edited June 6, 2016 by Palaxin

[wraitii]

I wouldn't use a function if I were you and just use global variables, depending on what you want to do. It's not like it's going to become a huge issue. Alternatively, you could return a tuple with all stuff in it. Or a dictionary, yes.

But again, not a python expert.

[Palaxin]

On ‎6‎/‎6‎/‎2016 at 5:44 PM, wraitii said:

I wouldn't use a function if I were you and just use global variables

I started to write classes, so the functions in one class can directly modify the variables of that class/object. The reason is because I want to bundle the majority of the code in a separate module/library which provides the necessary tools for heightmap input/output and manipulation.