Need help interpreting logs for bad crash. Was this caused by gyro drift?

ArduCopter Copter 3.3
1

Hello Everyone,

About a month ago I experienced a bad crash of my (clone) Pixhawk driven
quad. This crash resulted in the total destruction of the quad and also
nearly destroyed a $500 window.

The experience was very disconcerting for me, and I decided I would not fly
again until I had understood what went wrong.

Flight was in a field near a house. I was standing on the back porch. I
was practicing maneuvers in Stabilize mode and then at one point I heard an
EKF Error announced. A few moments later the quad started leaning towards
me and accelerated in my direction. It was non-responsive to controls at
this point. Fortunately it missed me but it slammed full speed into the
house, destroying itself and nearly taking out a brand new window. The
crash was hard enough to leave 1/2" dents in the siding of the house.

So what happened?

I have been looking at the log files since the crash, and feel that the
crash is possibly related to IMU.GYRY and IMU2.GYRY drifting apart by (what
appears to be) a small amount. Is this the gyro drift issue that others
had seen in the past? Or was this caused by aliasing due to some
high-frequency vibration which doesn’t seem to appear in the logs? Was
this caused by a poor quality gyro mounted on a clone Pixhawk?

Here are my notes:

Physical configuration

Platform is DJI F450 clone
Using clone Pixhawk (Ebay) with attached M8N GPS / Compass
Pixhawk mounted on Anti-Vibration plate
Pixhawk running APM:Copter V3.3.3
ESCs are Hobbysky XRotor 20A flashed with BLHeli (latest version)
Motors are Sunnysky x2212 KV980
Battery is 3S 4500
Props are 10"

TIMELINE

37:48 - IMU.GYRY and IMU2.GYRY drift apart for the first time
37:48 - EKF3.IMX,IMY,IMZ exceed recommended max of 50
37:52 - Start of Sharp spike in RCIN.C3. Spike ends one second later at
37:53 (pilot attempting correction?)
37:52 - We see a corresponding spike in MAG.MagX which goes from 200 to 400
37:53 - We see EKF3.IVN, IVE and IVD diverge.
37:54 - We see EKF4.SMX cross the 1.0 limit (Magnetometer inconsistency)
37:55 - EKF3.IVN, IVE and IVD go flatline

Output of Mission Planner’s automatic log file analyzer:

Test: Balance/Twist = NA -
Test: Brownout = GOOD -
Test: Compass = FAIL - Large change in mag_field (366.09%)
Max mag field length (596.92) > recommended (550.00)

Test: Dupe Log Data = GOOD -
Test: Empty = GOOD -
Test: Event/Failsafe = GOOD -
Test: GPS = FAIL - Min satellites: 0, Max HDop: 99.99
Test: IMU Mismatch = GOOD - (Mismatch: 0.66, WARN: 0.75, FAIL: 1.50)
Test: Parameters = GOOD -
Test: PM = NA -
Test: Pitch/Roll = NA -
Test: Thrust = NA -
Test: VCC = GOOD -

At this point I could really use some help in analyzing this log file. I
would very much like to know what exactly happened here. Additionally, I
am hoping this data can serve as a guide for the developers working on the
APM EKF code.

Here is the link to the log file:
https://dl.dropboxusercontent.com/u/26261321/2016-04-09%2016-36-10.bin

Thank you!
-Geoffrey

2

To be clear it’s not gyro drift but GPS drift. From the logs you expirenced the worst case I have seen. More than likely the trees to the south of the home caused the drift.

What happens is the GPS suddenly thinks that it is several meters away from it’s current position and tells the copter to move in the opposite direction to correct for this.

To see this for yourself open your log in Mission Planner and select Mechanical Failure from the drop down. Now click on the Show Map check box to see your position and the graph side by side. From there resize the graph so that its have the screen in the middle. Now double click on spots in the graph on the left and notice the tier drop move on the right. This is where the GPS thinks the copter is. As you click points closer to the end of the log you will notice that the copter flies off to the East which cause the copter to fly at you at full throttle.

Mike

3

Thanks Mike,

I was flying in Stabilize mode. I was under the impression that GPS errors wouldn’t cause flight issues like this when in Stabilize. Loiter yes but not Stabilize. Is that incorrect?

The reason I was flying in Stabilize was because I was concerned about the nearby trees shading the GPS signal.

Best Regards,
Geoffrey

4

Geoffrey,
You are correct, the FC does not use the GPS in stblz mode, at all, neither in AltH mode. Read wiki here - http://ardupilot.org/copter/docs/flight-modes.html

Unless you think you were in stblz mode, but in fact the copter was
in a GPS-based mode…

gG

5

I too like you was under the same impression but as of 3.3.3 they added it to Stabilize as well. They are working on a fix but as of right now it’s there. I believe you can turn it off by setting: EK2_ENABLE: Enable EKF2.

Mike

6

Just to confirm:

In 3.3.3 the GPS is used while in Stabilize mode?

That comes as a bit of a shock. I was under the impression that one of the points of Stabilize was that it was not dependent on the GPS. I’ve been training myself to switch from Loiter/Auto to Stabilize in the event of a GPS glitch so that I can attempt recovery.

7

Can we get confirmation from any developers that they have mixed in GPS with Stabilise mode.
This is quite a bug.
Stabilise was meant to be the base level of control when all auto sensors were going wrong.
GPS of all things, the variable and easily upset sensor on board is now mixed with stabilise.

Please tell me this is not so.

8

After some thought I believe that the GYRY delta between IMU and IMU2 may be the smoking gun in this crash.

Let’s for the time being assume that stabilize mode is not using the GPS.

The difference in gyroscope (GYRY) between IMU and IMU2 indicates to me that one of the IMUs has failed. It should not be possible for these gyroscopes to have a consistent mismatch like this, since the gyros are effectively measuring changes in rotational velocity.

This looks to me like one of the gyros has started producing some sort of offset in addition to the delta v it is supposed to be measuring. I can imagine that any sort of offset in the gyro output would wreak havok on the EKF as it is attempting to do estimate vehicle posture.

Thoughts?

9

When an IMU fails it usually flat lines. They stayed very consistant right up to the crash including during the EKF error. On the other hand the EKF error occurred just as the GPS drifted severally to the East as the copter pitched back and flew to the West.

Mike

10

Ok let’s assume it was the issue with GPS being used while in Stabilize mode.

Can you point me to the bug tracking this issue?

I looked through the bugs listed on Github and wasn’t able to find something that looked like a match for this issue.

Thank you!

11

I think this is it: https://github.com/ArduPilot/ardupilot/issues/2977
And another with Plane: https://github.com/ArduPilot/ardupilot/issues/4186

Mike