Multiple flyaways in guided - do I need to tune the EKF?

,

I had 3 successive flyaways yesterday while flying in guided mode. The copter darted towards the horizon at high speed until I regained control by switching to LAND mode.
In all 3 cases that happened after a couple of minutes of uneventful flight with the copter doing normal manoeuvers in guided mode. In flight the onboard computer is sending velocity commands to the pixhawk

Here is what I’ve found in the log for 1 of the flights:
The GPS speed log shows the flyaway very clearly (speeds up to 8m/s):

The EKF IVE velocity innovation spikes to 3 at the time of the flyaway:

Same thing for the EKF IPE which spikes at 6m during the flyaway.
The satellite count is 10 for the entire flight before the flyaway, it dips to 9 two seconds after the flyaway starts.
Here is the log:
2017-05-12 12-45-02.log (4.2 MB)
And the bin file:
2017-05-12 12-45-02.bin (1.9 MB)

I’ve flown this copter for 50+ flights but this is the first time that happens.
Main questions I’m trying to answer:

  1. What triggered the flyaway?
  2. How do I change parameters so that doesn’t happen again?
  3. If that ever happens again, how do I prevent losing my copter if it flies to the horizon? (bar retaking manual control) Will the geofence stop it? Should I set EK2_GLITCH_RAD to a lower value?

Thanks for the help!

You’re running copter 3.4.3 and there was a bug fix in copter 3.4.5 relating to a problem with guided mode. I’m not sure if this is relevant to your problem, but would suggest upgrading to 3.4.6 to be safe anyway.

You’ve also got your log bitmask set to 1854 so the logs are missing IMU and RC data which might make issues like this more difficult to diagnose.

Hopefully someone else can answer you questions about EK2_GLITCH_RAD and geofence.

Justin

Hey Justin, do you have a link to that bug? I want to take a look and see if that could have caused the flyaway.

Again, not sure if it’s relevant, but seeing as you’re using guided mode probably a good idea to upgrade anyway.

http://discuss.ardupilot.org/t/copter-3-4-5-released/14828

I just checked that bug. It doesn’t explain the flyaway problem but it explains a “crash after takeoff” problem I had last month when taking off in guided. I broke a few frames over this, thanks for pointing it out!

Regarding the flyaway my guess is something pushed the EKF innovations values through the roof… I am just not sure what.

It would be interesting to see vibration levels from this rig.

Set log_bitmask to 176126, do a quick test flight and post the log here. (Fly around rather than just hovering)

I’ve set the log_bitmask for vibration logging +EKF LOG_REPLAY=1 and I’ve done another test flight that ended up in a flyaway… so we’ve got data to work with!
The copter crashed really hard and the battery got disconnected on impact so the log just seems to end in mid air but basically the end of the curve is the crash. Still we have about 30s of flight where the copter made a few evolutions and logged data.

Because of LOG_REPLAY enabled the logs are quite big so I’ve uploaded them on Google Drive:
Here is the log in txt format:


And the log in bin format:

I see Vibe levels around 5, seems acceptable, what do you think?
On the other hand the EKF innovations are way beyond normal values (IVE and IVN peak at 40), even before takeoff this time, which seems very odd. Not sure what’s going on here. I need to run it the log through EKF Replay see if I can find what’s going on.

Hi Clem,

Rather than tuning the EKF, it’s usually best to find out exactly what’s upsetting it in the first place.

Both your logs have the following entries just after take-off

EKF2 IMU0 ground mag anomaly, yaw re-aligned
EKF2 IMU1 ground mag anomaly, yaw re-aligned

Seems other people experienced similar issues:

http://discuss.ardupilot.org/t/ekf-ground-mag-anomaly-has-anyone-heard-of-it/15665

and more here…

http://discuss.ardupilot.org/t/nightmare-compass-calibration-solved/15421

I can’t really tell exactly what is happening, but if this was my rig I’d have the compasses as my prime suspects.

Justin

Hey Justin, I ended up finding the culprit. Right before the flyaways the GPA.SAcc parameter suddenly starts degrading in the logs. It was a GPS issue!