Servers by jDrones

Overcoming Lua floating point math precision errors (latitude/longitude)

I was writing a script to string multiple waypoint missions together by loading them sequentially from the SD card when I noticed a peculiarity in the example script method for loading waypoints.

The example scripts read the text files line by line, using a table (called data) to store each set of waypoint parameters. The latitude/longitude integer values are set like this (where data fields 9 and 10 are in floating point decimal degrees):


Unfortunately, that’s only good to about 5 decimal places (~1m accuracy), and waypoints for my use case require near centimeter accuracy - the full 7 decimal places.

Fortunately, it appears that you can overcome the issue by simply manipulating the string values, like this:

-- the AP Lua implementation appears very susceptible to floating point math errors
-- this function takes a decimal number in string format and returns an integer
-- representation of its value * 10^7 (as in ArduPilot lat/lng arguments)
-- without using any floating point math (takes *much* longer)
function strf_to_stri(x)
  local whole, fraction = x:match("(.+)%.(.+)")
  if (fraction == nil) then
    whole = x
    fraction = '00000000'
  if (string.len(fraction) > 8) then
    fraction = fraction:sub(1, 6 - string.len(fraction))
  fraction = fraction .. string.rep('0', 8 - string.len(fraction))
  return whole .. fraction:sub(1, -2)

If you’re about to say something like, “But Yuri, just use string.format()!,” I tried that. Same precision error. :smiley:

I should probably lop off the 8th fractional character a little sooner and improve the overall efficiency of the function. I was accounting for the fact that the typical input to the function is expected to have 8 decimal places (as in Mission Planner’s saved waypoint files).

1 Like
Servers by jDrones