Servers by jDrones

# Overcoming Lua floating point math precision errors (latitude/longitude)

I was writing a script to string multiple waypoint missions together by loading them sequentially from the SD card when I noticed a peculiarity in the example script method for loading waypoints.

The example scripts read the text files line by line, using a table (called `data`) to store each set of waypoint parameters. The latitude/longitude integer values are set like this (where data fields 9 and 10 are in floating point decimal degrees):

``````item:x(data[9]*10^7)
item:y(data[10]*10^7)
``````

Unfortunately, thatâ€™s only good to about 5 decimal places (~1m accuracy), and waypoints for my use case require near centimeter accuracy - the full 7 decimal places.

Fortunately, it appears that you can overcome the issue by simply manipulating the string values, like this:

``````-- the AP Lua implementation appears very susceptible to floating point math errors
-- this function takes a decimal number in string format and returns an integer
-- representation of its value * 10^7 (as in ArduPilot lat/lng arguments)
-- without using any floating point math (takes *much* longer)
function strf_to_stri(x)
local whole, fraction = x:match("(.+)%.(.+)")
if (fraction == nil) then
whole = x
fraction = '00000000'
end
if (string.len(fraction) > 8) then
fraction = fraction:sub(1, 6 - string.len(fraction))
end
fraction = fraction .. string.rep('0', 8 - string.len(fraction))
return whole .. fraction:sub(1, -2)
end
``````

If youâ€™re about to say something like, â€śBut Yuri, just use `string.format()`!,â€ť I tried that. Same precision error.

I should probably lop off the 8th fractional character a little sooner and improve the overall efficiency of the function. I was accounting for the fact that the typical input to the function is expected to have 8 decimal places (as in Mission Plannerâ€™s saved waypoint files).

1 Like
Servers by jDrones