Overcoming Lua floating point math precision errors (latitude/longitude)

Yuri_Rage · September 6, 2021, 5:49pm

I was writing a script to string multiple waypoint missions together by loading them sequentially from the SD card when I noticed a peculiarity in the example script method for loading waypoints.

The example scripts read the text files line by line, using a table (called data) to store each set of waypoint parameters. The latitude/longitude integer values are set like this (where data fields 9 and 10 are in floating point decimal degrees):

item:x(data[9]*10^7)
item:y(data[10]*10^7)

Unfortunately, that’s only good to about 5 decimal places (~1m accuracy), and waypoints for my use case require near centimeter accuracy - the full 7 decimal places.

Fortunately, it appears that you can overcome the issue by simply manipulating the string values, like this:

-- the AP Lua implementation appears very susceptible to floating point math errors
-- this function takes a decimal number in string format and returns an integer
-- representation of its value * 10^7 (as in ArduPilot lat/lng arguments)
-- without using any floating point math (takes *much* longer)
function strf_to_stri(x)
  local whole, fraction = x:match("(.+)%.(.+)")
  if (fraction == nil) then
    whole = x
    fraction = '00000000'
  end
  if (string.len(fraction) > 8) then
    fraction = fraction:sub(1, 6 - string.len(fraction))
  end
  fraction = fraction .. string.rep('0', 8 - string.len(fraction))
  return whole .. fraction:sub(1, -2)
end

If you’re about to say something like, “But Yuri, just use string.format()!,” I tried that. Same precision error.

I should probably lop off the 8th fractional character a little sooner and improve the overall efficiency of the function. I was accounting for the fact that the typical input to the function is expected to have 8 decimal places (as in Mission Planner’s saved waypoint files).

sail4.5 · December 23, 2022, 8:18pm

Thanks for this tip Yuri. I’m not sure exactly where to insert the function into the code that is reading the waypoint table. If I try to convert the data read for index 9 or 10 (GPS coordinates) I get an error “attempt to index a number value (local ‘x’)”.

Yuri_Rage · December 23, 2022, 8:25pm

Give me a code snippet to work with. Can’t help with that vague description.

sail4.5 · December 23, 2022, 8:48pm

local data = {}
for i = 1, 12 do
new_read = file:read(‘n’)
if i == 9 or i == 10 then
data[i] = strf_to_stri(new_read)
else
data[i] = new_read
end

  if data[i] == nil then
    if i == 1 then
      gcs:send_text(6, 'loaded mission: ' .. file_name)
      return -- got to the end of the file
    else
      mission:clear() -- clear part loaded mission
      error('failed to read file')
    end
  end
end

Yuri_Rage · December 23, 2022, 9:22pm

You have a type conversion issue.

An easy fix is:
data[i] = strf_to_stri(tostring(new_read))

sail4.5 · December 23, 2022, 10:54pm

ah … thanks so much!

regarding this process of loading missions from scripts, it took me awhile to realize the process was working (actually loading wp’s onto the ardupilot) because Mission Planner was not showing any of the waypoints. Is there a way to refresh the map on Mission Planner to show the loaded waypoints?

Yuri_Rage · December 23, 2022, 11:18pm

There’s a button to read the mission from the autopilot.

Too many people confuse “Mission Planner” for their autopilot and its internal features. It is just the GCS. It only knows what comes down from telemetry.

sail4.5 · December 23, 2022, 11:35pm

Thanks again.
Still not sure if I’m getting the right results from this code.
For example, my last waypoint is this:
|2|0|3|16|0.00000000|0.00000000|0.00000000|0.00000000|28.60903828|-81.26689837|0.000000|1|
my code says:
if i == 9 or i == 10 then
data[i] = strf_to_stri(tostring(file:read(‘n’)))
else
data[i] = file:read(‘n’)
end

28.60903828 produces your fraction value of 60904 and a final output of 286090400
-81.26689837 produces your fraction value of 2669 and a final output of -812669000

it seems like file:read(‘n’) is rounding the waypoint value before it gets to your function

Yuri_Rage · December 24, 2022, 12:33am

Ah yes! Reading as a number is problematic. You need to read those tokens as strings.

This example does that using gmatch():

github.com

ArduPilot/ardupilot/blob/master/libraries/AP_Scripting/examples/mission-edit-demo.lua

-- mission editing demo lua script.
-- by Buzz 2020
current_pos = nil
home = 0
a = {}
demostage = 0 
eventcounter = 0

function update () -- periodic function that will be called
    current_pos = ahrs:get_location()

  -- adds new/extra mission item at the end by copying the last one and modifying it
  -- get number of last mission item
  wp_num = mission:num_commands()-1
  --get last item from mission
  m = mission:get_item(wp_num) 
  -- get first item from mission
  m1 = mission:get_item(1) 

  if wp_num >  0 then

This file has been truncated. show original

sail4.5 · December 27, 2022, 9:46pm

Thanks Yuri, and Happy Holidays!

I have the waypoint import working, however the coordinates in Mission Planner are different from what was sent to it. I’ve uploaded a picture showing a screen capture from the Message Tab vs the coordinates on the Planner screen. Not sure what to make of it.

Yuri_Rage · December 28, 2022, 3:40am

You’re still doing arithmetic at too many decimal places, as far as I can tell.

Again, no real help without seeing the code itself.

sail4.5 · January 3, 2023, 1:34am

You were right, I was still doing too many decimals. Fixed that and can import an 8 decimal number using your function.

However there is a problem that I can’t resolve. If I provide a waypoints file with 7 decimal coordinates, neither the mission_load.lua nor the mission_edit lua retrieves the coordinates properly (unless I use your function). I have attached my waypoints text file so you can test this. This isn’t a rounding issue or precision issue, the imported data just doesn’t make sense. If I can save a waypoints file using Mission Planner why can’t I retrieve it using Lua?
MM_#01.txt (336 Bytes)

Yuri_Rage · January 3, 2023, 1:37am

You literally just said “unless I use your function.” That’s why it’s there.

It’s a floating point precision error in the Lua interpreter, plain and simple. Import as a string. Remove the decimal character. Convert to integer. That’s the workaround.

sail4.5 · January 3, 2023, 4:23pm

Fair enough. A good solution for retrieving coordinates from a waypoint file.

The underlying issue seems to be the Lua interpreter, which only handles a maximum of seven digits. Anything more than that is pruned off. This makes it impossible to do math calculations with precision using the Lua interpreter. This applies to any math, not just coordinate calculations.
In the following example, b = tonumber(a) where ‘a’ is a twelve character string.

Let me know if I am misunderstanding the situation.

Thanks Yuri.

Yuri_Rage · January 3, 2023, 4:28pm

You’re not misunderstanding. I’d have to do a little math to determine the true limit, but I suspect signed floating point numbers are stored in 32 bits or less, making them permanently imprecise. An alternative is to do unsigned 32 bit integer math (uint_32t objects), allowing for much larger absolute magnitudes. ArduPilot’s C++ codebase uses this tactic frequently to maintain high degrees of precision without the costly requirements of 64-bit arithmetic operations.

EDIT:
Rather than doing the math myself, here’s an article that describes 32-bit floating point precision:

Demystifying Floating Point Precision (demofox.org)
A float has 23 bits of mantissa, and 2^23 is 8,388,608. 23 bits let you store all 6 digit numbers or lower, and most of the 7 digit numbers. This means that floating point numbers have between 6 and 7 digits of precision, regardless of exponent.

sail4.5 · January 3, 2023, 8:04pm

Thanks once again for your quick and thorough reply Yuri, much appreciated.

sail4.5 · January 18, 2023, 2:50pm

Hi Yuri,

I know this question is not related to the topic, but related to scripting in general (wasn’t sure how to contact you directly)

I put a script in ROMFS and now I don’t know how to remove it. It was a test script and now whenever I enable scripting it starts running in the background. Very frustrating.

I was able to solve the problem (I thought) by loading a different airframe and the then going back to a clean rover install. I then loaded saved parameters to get back to my original setup for the rover. However when I turn scripting on that test script reappears. It is not in any of the MAVftp directories so must be in ROMFS

iampete · January 18, 2023, 2:57pm

Use the parameter SCR_DIR_DISABLE to disable scripts in ROMFS. The only way to remove is by re-flashing firmware.

sail4.5 · January 18, 2023, 2:57pm

thanks for the quick response Peter!

sail4.5 · January 19, 2023, 3:27pm

Hi Peter,
I have disabled the scripting and re-flashed the firmware by loading a different airframe and then rel-loading the rover. Is there a better way to reflash the firmware?

Even with the scripting disabled, this phantom script keeps showing up. I have uninstalled and re-installed Mission Planner as well (latest version),

Any thoughts on how to fix this?