I had a serious issue during a BVLOS flight where the EKF position departed drastically from the GPS position. An anonymized (locations changed) logfile can be found at:
Apologies I can’t upload the BIN file with the actual location.
The first minute of the flight was normal - at 75 seconds into the flight, the GPS.Lat and POS.Lat diverge drastically, and the same with GPS.Lng and POS.Lng. The reported positions were on a straight line heading approximately upwind when in fact the aircraft was heading rapidly downwind. I had to manually fly the aircraft back using FPV and the positions did not correct themselves until well after landing.
The wind was about 15m/s - within the capabilities of the platform normally.
The GPS lock was excellent - 28 to 30 satellites, HDoP < 0.6 at all times. The GPS is a HolyBro F9P operating in non-RTK mode. The flight controller is a Holybro PixHawk 4 Mini.
I have never seen the EKF output depart from the GPS location so significantly and think that this is a serious bug that really needs fixing to prevent fly-aways, (RTL would not work) whether or not there were contributing causes. If the EKF is very unhappy, the reported position really ought to follow a good GPS fix unless there is a very good reason why not (can’t think of one).
There are two hardware issues that could potentially have contributed
Compass interference from the motor current, which although they have previously caused no issue beyond occasional “compass inconsistent” warnings, I had disabled compass 2 (it is closer to the power wiring and has worse interference) and I’d attempted to improve the interference with COMPASS_MOT_* parameters based on ground full-power tests. I had overlooked to change COMPASS_MOTCT however. I will of course be seeking to physically improve this interference by increasing separation in future. The wiki documentation for COMPASS_MOT_X etc could benefit from improvement by someone that knows how it works, especially around determining values.
Some unexpected rain starting almost immediately after takeoff may have affected the air speed sensor. I have had issues with rain causing incorrect airspeed readings on previous flights but have not noticed any obviously wrong air speed readings in this log file, and wrong airspeed readings have never caused incorrect position previously either.
Edit: I forgot to mention, I noted two messages in the log about 8 seconds before the positions drastically diverged: “EKF2 IMU0 yaw align to GPS velocity” and “EKF2 IMU1 yaw align to GPS velocity”. I note a small jump in POS.Lng at this time but the general trend continues to follow GPS for about 8 seconds.
For some reason the scale of some of the values in the log file aren’t working well for me tonight. I don’t know if it’s the log file or my computer.
There are certainly some compass issues, as you’ve pointed out. That wiring will need to be cleaned up or the antenna moved. That’s probably causing a lot of the trouble.
15 m/s is a pretty healthy wind, and I think it may have been pushing the limits of the PID loop. In general the PID loop looks like it isn’t tuned very well as the actual and desired pitch/roll do not track closely. If the plane is properly tuned then this may be a sign that 15m/s is too much for the system to keep up. There’s a few moments of significant oscillation, mostly in the roll. It’s probably worth making sure the tune is dialled in, or at least verifying it under better conditions.
There is a vibration that tracks throttle inputs. Possibly minor propeller damage. Not significant, but it’s there.
Thanks Allister, I’ll double check the prop and motor shaft. It’s a folding prop so a bit trickier to balance but I had done my best.
I’ll look again into the tuning from other log files and try another autotune. I think I transferred the parameters from an identical airframe so it’s possible there are some differences on this one.
Even with a terribly set up airframe, I don’t think the EKF output should ever be giving positions so divergent from the GPS position though…
Yes, it’s a fairly strong wind but certainly perfectly flyable and the autopilot has always performed sensibly in these winds over many hours of flight to date.
Thanks Allister, I’ll do this next time I’m able to fly, and improve the wiring/compass proximity as much as possible.
Looking at the parameters, the value of EK2_GLITCH_RAD was set to 25, which is the default, I believe. If I’ve understood this correctly, this means that the EKF position output should never exceed 25m from the GPS position. It certainly exceeded it in this case!
Regardless of how well or badly the PID loops and compass were working, from my limited understanding, I still think there may be a fundamental bug here that needs fixing. I suspect the wind is not the cause of the issue. It’s not that the aircraft can’t navigate effectively, it’s just completely wrong about where it is, so it cannot navigate at all! If I’d left in auto or RTL (or had a radio failure) I think it would have continued to head downwind at high speed.
With a bit of further digging into the logs, I have noticed that the various *.Yaw values behave differently at the exact time that the GPS.Lat and POS.Lat (and Lng) diverge:
AHR2.Yaw : shows a jump from 238 to 308 degrees (jumps to value of GPS.Gcrs)
NKF1.Yaw : no jump seen, value ~224
NKF6.Yaw : no jump seen, value ~218
ATT.Yaw : switches from tracking NKF1.Yaw to NKF6.Yaw
GPS.Gcrs : no jump seen, value ~308
GPS.GCrs is of course expected to differ significantly from actual yaw due to the wind.
I can’t find much documentation about the meaning of AHR2.Yaw (Backup AHRS data?) or ATT.Yaw (Canonical Vehicle Attitude?), but I found something suggesting that NKF1.Yaw (EKF2 estimator outputs?) is primary EKF2 yaw solution, and NKF6.Yaw is secondary EKF estimation, so perhaps the problem isn’t in the EKF? Does anyone know what the exact meaning of these are?
For what it’s worth, I can’t see anything particularly different about the MAG.MagX/Y/Z, gyro or accelerometer values around the time of the divergence.
I’m not really qualified to investigate why the EKF has gotten confused especially on Planes but the NKF3 “IVN” and “IVE” (velocity innovations for north and east) flatline three times which I think indicates that the EKF was no longer attempting to fuse in velocities.
that graph shows a huge change in Y magnetic field when you do a motor test before takeoff.
It should have recovered though, as the EKF gives up on the compass and aligns to GPS velocity.
The weirdest part of the log is the discrepancy between position innovations and the actual position. Position innovations are around 300m
but difference between GPS position and EKF position gover over 4km. That makes no sense. Velocity fusion also stops.
What tool did you use to anonymise the log?
Yes, I’d used the motor tests from previous flights to come up with the values for COMPASS_MOT_* however I failed to spot the COMPASS_MOTCT parameter that needed turning on. (Also not sure whether I got the values the correct polarity i.e. +/- value.) I do intend to increase the physical separation between the power wiring and the compass in the near future.
I used the “anon log” function in the latest beta of Mission Planner.
The graphs for the positions look exactly as in the non anonymised BIN file except for the expected shift. There really is a >4km difference! It was extremely unnerving as the GPS position isn’t relayed to the ground station. A good learning experience for me on when to disregard the telemetry and trust your eyeballs on the FPV. I’ll be quicker to recover next time!
I can confirm that the graphs above (NKF3.IPN and NKF.IPE, also NKF3.IVE and NKF3.IVN) look exactly the same as the BIN file.
Thanks for the input. The airspeed maximum is sensible but at its lowest, it is perhaps slightly higher than usual at lower throttle values without loss of height. However given the rapidly changing bearing, and the strong wind, it may indeed be perfectly accurate. Looking at the difference between GPS speed and airspeed given the wind speed, the readings certainly aren’t totally unbelievable.
I’ve just had this happen again, the next flight after the above, having moved the external compass well away from all power wiring and having upgrading firmware to 4.0.8. This time the issue occurred approximately 13 minutes into the flight. The maximum size of the error was about 6km. Once again, following landing, (about 45 seconds in this case) the position jumped back to match GPS.
I had several successful flights with this aircraft with no issues before the first flight with this problem. I may try swapping out the PixHawk 4 Mini and the Holybro F9P GNSS Rx if I can lay hands on replacements. I will also try doing some autotune I think.
No, there was a clear view of the sky. The GPS signal is always good in the logs and the positions accurate, and hdop good.
I did an autotune, which did improve flight slightly. However the issue continues to occur on some flights. I haven’t yet been able to swap out the hardware.
An identical aircraft with a Holybro M8N receiver has never had this issue, flying in the same conditions. I have got tens of hours on this. It’s reliable.
Has anyone else used an Holybro F9P with Arduplane without issues?
