Terrain generation issue

Hi everyone,

We are trying to use the new create_terrain.py python script to pre-load the ‘NxxExxx.DAT’ terrain files onto a Pixhawk so that we do not need to upload terrain height information during a flight. However, we found that once we had copied the files to APM/TERRAIN folder on theSD card, ArduPlane was overwriting them with incorrect data. As a further test, we deleted the terrain files and let ArduPlane generate the .DAT files by requesting terrain information via Mavlink. It appears that these files also contain errors.

Some details regarding our setup:
- ArduPlane 4.0.5
- GCS: Mission Planner

I wrote a python script to visualize the terrain files that are being generated by the create_terrain.py as well as by ArduPilot to demonstrate the issue:

Using our location as an example S35E018, we obtained two S35E018.DAT files, one generated by the create_terrain.py script and the other from ArduPlane (with an initially empty APM/TERRAIN folder). Going through the lat/lon data contained in the S35E018.DAT file generated by ArduPlane, it seems like most of the blocks being loaded in are from 1 longitudinal degree East of the correct tile (S35E019), and some of the smaller groups of blocks are being loaded from 1 latitudinal degree north of the correct location (S34E018).

Here is the plot of the S35E018.DAT file generated by the script (which is correct):

Here is the plot of the S35E018.DAT file generated by ArduPlane (which contains errors):

For anyone else wanting to better understand the terrain file structure, here is a diagram of what I roughly understand the structure to be (there may be mistakes in here so please let me know if there are changes needed):

There is a bug with terrain file handling in that version (as your pictures show). It is fixed in ArduPlane 4.0.6 and later.

There is a terrain file server over at https://terrain.ardupilot.org/ which has the correct tiles. You can use this to pre-load tile on to the SD card.

EDIT:
I don’t think 4.0.6 has been released quite yet.

Thanks @stephendade, I will have a look at 4.0.6.

For those interested, here’s an explanation and PR for the terrain bug: https://github.com/ArduPilot/ardupilot/pull/14254

It looks like ArduPlane 4.0.6 is no longer overwriting the files that are copied to the SD card. However, when the SD card is cleared and ArduPlane generates the .DAT files, it seems to be generating some unreadable blocks.

My understanding is that the .DAT files consist of enough 2048-byte blocks to cover the 1 lat/lon degree for the file, where each 2048-byte block consists of the following data:

TYPE	        NAME	    BYTES
---------------------------------------
uint64_t        bitmap      [0:8]
int32_t         lat         [8:12]
int32_t         lon         [12:16]
uint16_t        crc	        [16:18]
uint16_t        version	    [18:20]
uint16_t        spacing	    [20:22]
int16_t[28x32]  height	    [22:1814]
uint16_t        grid_idx_x  [1814:1816]
uint16_t        grid_idx_y  [1816:1818]
int16_t         lon_degrees [1818:1820]
int8_t          lat_degrees [1820:1821]
   trailer of 227 bytes     [1821:2048]

When the .DAT files are generated by ArduPlane, are they simply meant to contain 2048-byte blocks or is there other information that is included in these files? They don’t seem to be generated in the same way as the create_terrain.py script as, for instance, the blocks generated by ArduPlane are not ordered. I was finding that the first 48 2048-byte blocks in the file that was generated by ArduPlane 4.0.6 were invalid; however, the remaining blocks in the file were correct. Is this perhaps some header information?

@LiamClarkZA Do you by chance still have the plot_terrain.py file somewhere?
Also your graphic is really useful and deserves to be on the wiki somewhere!

Hello @LiamClarkZA, @hendjosh

I’m looking for the plot_terrain.py file too… is it available somewhere? I couldn’t find it in the github repo fork…

Thanks a lot!

Hello this is a similar script that saves each .dat file as a jpeg image however there is no crc handling or missing data handling.
Hope it helps those trying to write their terrain protocol handling.

import os,sys,struct
import math
import numpy as np
import sys
from PIL import Image



TERRAIN_GRID_MAVLINK_SIZE = 4
# MAVLink sends 4x4 grids
# a 2k grid_block on disk contains 8x7 of the mavlink grids.  Each
# grid block overlaps by one with its neighbour. This ensures that
# the altitude at any point can be calculated from a single grid
# block
TERRAIN_GRID_BLOCK_MUL_X = 7
TERRAIN_GRID_BLOCK_MUL_Y = 8


IO_BLOCK_SIZE = 2048

TERRAIN_GRID_BLOCK_SIZE_X = (TERRAIN_GRID_MAVLINK_SIZE*TERRAIN_GRID_BLOCK_MUL_X)
TERRAIN_GRID_BLOCK_SIZE_Y = (TERRAIN_GRID_MAVLINK_SIZE*TERRAIN_GRID_BLOCK_MUL_Y)


def geo2dist(coords_1,coords_2):
	R=6471 * 10**3
	phi_1 = (coords_1[0]*np.pi)/180.00
	phi_2 = (coords_2[0]*np.pi)/180.00
	d_phi = phi_1-phi_2
	d_lbd = (coords_1[1] - coords_2[1])*np.pi/180.00
	a = (np.sin(d_phi/2))**2 + np.cos(phi_1) * np.cos(phi_2) * (np.sin(d_lbd/2))**2
	c = 2* np.arctan((a**0.5) / ((1-a)**0.5))
	dist2wp = R * c
	return dist2wp

def to_float32(f):
	return struct.unpack('f', struct.pack('f',f))[0]

class Terrain_Manager:
	def __init__(self,filename):
		self.filename = filename
		self.fh = open(self.filename, 'rb')

		self.fh.seek(0, os.SEEK_END)
		self.file_size = self.fh.tell()
		self.total_blocks = self.fh.tell()/IO_BLOCK_SIZE
		print("Total blocks:	", self.total_blocks)
		self.fh.seek(0, os.SEEK_SET)

		self.lat_count = []
		self.lon_count = []
		self.RESOLUTION = 0.01
		self.parsed_data = []
		lat_array=[]
		lon_array=np.array(0)
		lat_row_array = np.array(0)
		full_file_array=np.array(0)
		self.fh.seek(0, os.SEEK_SET)
		image_height  = 7*4 - 1
		image_width  = 8*4 - 1
		i_g = 0 
		prev_lat = 0
		prev_lon = 0
		first_block_parsed_in_row = False
		for gridblockindex in range(int(self.total_blocks)):
				
				geo_raster = np.zeros((8*4,7*4,3),dtype=np.uint8)
				file_offset		= gridblockindex*IO_BLOCK_SIZE	

				self.fh.seek(file_offset)
				buf = self.fh.read(IO_BLOCK_SIZE)
				(bitmap, lat, lon, crc, version, spacing) = struct.unpack("<QiiHHH", buf[:22])
				if bitmap!=0:
					(heights) = struct.unpack("<896h", buf[22:1814])
					ht_index=0
					for gy in range((7*4)):
						for gx in range((8*4)):
							gx_pos=gx
							gy_pos = gy
							R_MARKER = 256*256
							G_MARKER = 256
							B_MARKER = 1
							R_HEIGHT = ((heights[ht_index])/self.RESOLUTION)
							geo_raster[ gx_pos, gy_pos , 0] = int((R_HEIGHT - R_HEIGHT%R_MARKER)/R_MARKER)
							G_HEIGHT = R_HEIGHT%R_MARKER
							geo_raster[ gx_pos, gy_pos , 1] = int((G_HEIGHT - G_HEIGHT%G_MARKER)/G_MARKER) 
							B_HEIGHT = G_HEIGHT%G_MARKER
							geo_raster[ gx_pos, gy_pos , 2] = int((B_HEIGHT - B_HEIGHT%B_MARKER)/B_MARKER) 
							ht_index+=1

					if prev_lat!=lat and first_block_parsed_in_row:
						print("New row ",lat,lon)
						# lat_row_array = geo_raster
						lat_array.append(lat_row_array)
						lat_row_array = geo_raster
						print(lat_row_array.shape)
					if not first_block_parsed_in_row:
						lat_row_array = geo_raster
						first_block_parsed_in_row=True
						print("First block parsed",lat,lon)
						print(lat_row_array.shape)
					if prev_lat==lat and first_block_parsed_in_row:
						lat_row_array =  np.concatenate((lat_row_array,geo_raster[3:,:]),axis=0)
						print("concat along latitude",lat,lon)
						print(lat_row_array.shape)

					prev_lat=lat
		fg=0
		for i in (lat_array):
			if fg!=0:
				full_file_array =np.concatenate((full_file_array,i[:,3:]),axis=1)
			else:
				full_file_array = i
			fg+=1
			print("lat array index",i.shape,fg)


		im = Image.fromarray(full_file_array).convert('RGB') 
		im.save(filename + "test"+"_rgb"+".jpeg")


if __name__ == '__main__':
	for i in range(1,len(sys.argv)):
		print(sys.argv[i])
		try:
			z = Terrain_Manager(sys.argv[i])			
		except:
			pass

1 Like