Testing some autonomous missions this week I experienced a GPS glitch which caused the aircraft to have a massive position offset (actual location vs location the GPS thought it was).
I am trying to work out from the logs what has caused this. So far I can see nothing that would explain what caused the glitch. Need to understand if this is a one in a hundred occurrence or if there is something else going on here.
The track map clearly shows where things go wrong. The aircraft continues to fly the mission shape but thinks its 100 meters away from where it really is, it then begins to drift and loose altitude before I flip it into loiter and manually fly it home.
The glitch starts when for a split second there appear to be zero sats.
Interesting. This means that the error certainly lies with the GPS. I didn’t see any power-related problems, but something caused the GPS to lose its lock and then get this big offset. I see you’re using RTK (F9?), so I can only guess that the corrections might have changed somehow. Did something change on your base GPS?
We are indeed using the f9p with rtk and love it! Apart from this glitch it’s leaps ahead of the old m8p unit it replaced.
We are using a web based base station from Leica geosystems so it should be pretty good. There’s nothing in the server logs that suggest anything odd at that point.
During our testing that day we did experience issues with our Rfd radio dropping out occasionally. This appears to be down to it not liking sharing a power supply with a newly fitted camera unit. It is possible that the radio glitched, gave incorrect rtcm rtk messages and caused the weird offset we are seeing… but that seems really far fetched given that you’d expect it to drop back to 3dgps if the rtcm messages are delayed rather than jumping it’s position somewhere new. Is there any sanity check on the incoming rtcm stream in ardupilot?
The only thing I can think of that might cause this is the Pixhawk sending config changes to the GPS receiver; I’ve seen uBlox receivers momentarily lose all sats while being configured. It might possibly explain the offset as well. I consider this to be quite unlikely, though, because config changes are only sent when something changes or is different than the current configuration, and your auto_config and save_cfg parameters should have taken care of that long before you took off. That said, F9P integration is new, and I don’t know how well proven its driver is, so maybe there’s a bug.
I can at least explain the drone’s behavior, given the GPS offset:
When the offset happened, this triggered a GPS glitch, which just means that there is a disagreement between the EKF’s position estimate and the GPS readings. In the glitch state, the drone continues normally if nothing else is wrong, but eventually (after 10 seconds, iirc) the EKF will reset its position to the current GPS position in order to clear the glitch. Once the position estimate was reset, the navigation controller started flying it back towards the intended flight path, which is the southeast excursion you observed, since the GPS offset was telling it that it was too far northwest.
I agree with your explanation of the behavior. It makes sense and is nice to see things work as they should given a failure.
We are running 3.60 I believe the latest release has specific support added for the f9p. Do you have any idea what exactly has been added in the way of support?
The main F9 support patch is this one, which seems to mostly just detect it and ensure that we send the correct messages to it. Everything else is likely handled by existing uBlox GPS methods already used by other receivers. Looking at the Ardupilot repo, it seems that most F9-related PRs have been included in 3.6.10, except this one, which is only implemented in master. It looks like that one is just a bugfix for the timestamp, though.
So, it is probably unlikely that this was caused by some quirk of the F9 module specifically. I’m really out of ideas on what might have caused this. I think @WickedShell is experienced with GPS implementation, so maybe he has an idea.
