Interesting log: Hexa motor failure, undetected "crash" and a dizzy EKF2

Hi all,

at thorstenbehrens.de/HexMotorFailure_U.zip you find a log from a colleague of mine.

During an auto mission one motor stopped working. The reason must be a mechanical or some ESC related issue. Anyway, the log might be interesting since the hexa survived the motor failure with only some very small injuries.

Somehow the copter has not lost yaw control, i.e. no spinning, but just descended. In this respect I am wondering what caused CTUN.DAlt to decrease?

After some flight mode changes (in panic I guess) and a continuous descent the copter touched ground. Relatively hard. It lost parts of the payload resulting in a shift of the COG. I thought that such a hard “landing” should trigger the crash detection.

However, the motor, which failed came alive again, and due to fact that throttle was still high the copter took off again. During this period something must have happened to the EKF2 estimates, because a GPS glitch was detected. However, the number of sats and the HDOP remained constant. There is a single peak in pitch, but since HDOP did not change I think this is not the cause. Any idea?

Kind regards,
Thorsten

I can answer your questions regarding D.Alt and the crash detector.

This graph shows altitude, desired altitude, and throttle output. After the motor failure, the throttle output saturates at 1. This means that the copter did not have enough power to hover with a failed motor (actually it is more like 2 failed motors, since the flight controller must stop using the opposite motor in order to remain balanced). This caused the copter to start descending. As you can see in the graph, the altitude sinks below desired altitude. The reason that desired altitude also decreases is that it is being “dragged down” by the altitude. This behavior is a safety measure to make sure that there is never a huge difference between desired and actual attitude (also applies to roll, pitch, yaw), and this flight is a perfect example why: if the desired altitude stayed at 50 meters, the copter would fly up 50 meters as soon as the failed motor restarted!

The crash detector causes the copter to disarm which could be very dangerous if the copter is in flight, so the crash detection must be very conservative. This means that relying on the IMU’s to detect a crash by sensing a hard jolt could result in false detections. The following conditions must be true for two seconds for the crash detector to trigger:

  • Copter is inverted (desired vs. actual attitude is >30 degrees)
  • Copter is not accelerating (e.g., it’s lying on the ground)

Your last question is more difficult, so I’ll see if I can help with that when I get some more time to review the log.

1 Like

@Anubis thanks for having a look!

I did not knew that the DAlt is “dragged down” by the altitude. Makes sense, at least for altitude.

Regarding the crash detection I was aware of the conditions. This is why I put crash and landing in quotation marks. Because it was not a crash in the classical sense. However, there was clipping and heavy vibes close to the ground, which is obviously a pretty good indicator for a crash. So maybe some measure that accounts for such events should be added to the crash checks.

You can see that it was motor 2 when looking at RCOU.C2. The FRAME_TYPE is 0:Plus so it was the rear motor.
I thought that a hexa should start spinning and not descending. This is why I thought some devs might be interested in looking at it. On the other hand, it makes sense. The front motor tries to keep the balance and the other 4 motors produce the lift. They are not saturated. But providing more power would maybe result in a spin. So perhaps it is an effect the general power and weight of the system how it behaves in case of a motor loss.

Regarding the GPS glitch:

This is a graph of the EKF test ratios (red greed blue), which is a ratio of how much the sensor data deviates from the EKF estimate and the maximum acceptable deviation. Any test ratio that exceeds 1.0 is will cause the filter to reject the measurement.

For example, NKF4.SV (red) is GPS velocity test ratio. As you can see, the ratio exceeds 1 when the copter hits the ground. The yellow and orange lines are the EKF’s vertical velocity estimate and the GPS velocity measurement. They deviate by an amount that is greater than the maximum deviation EK2_VEL_I_GATE*. This prompts the flight controller to produce a GPS glitch error (ERR: GPS-2) and switch to the alternate EKF solution (ERR: EKF_PRIMARY-1). In this sense, a “dizzy EKF” is an accurate description. Later, when the copter starts flying again, the GPS error is cleared because the filter recovers from its concussion.

*Actually, vertical velocity alone might not have exceeded the maximum deviation. The test ratio is a combination of all the GPS speeds (north, east, down) which, when combined, exceeded the deviation.

Thanks Rick for the detailed explanation!
So in the case of a fast succession of a clipping/heavy vibes event, especially when the copter is close to the ground (above home), and the occurence of some dizzyness, it might be good to switch to land mode as a failsave option.