EKF failsafe and crash on Copter 4.0.7

I had a ~20kg X8 multicopter running a PixHawk 2 go into the sea recently. I only have the TLOG from the ground station for fairly obvious reasons! :frowning:

https://drive.google.com/file/d/1kaZValN-_Uc_QI5Rd76Db5UH3Pn88MBp/view?usp=sharing

I’d really like to understand what went wrong and how to avoid having the same thing happen again in future, and wondered whether someone with more experience of log analysis could take a look please? I have anonymized the log, so the actual location is different. The flight was over the sea.

The copter was successfully following waypoints, although there were amber EKF warnings and “bad gyro health” warnings on Mission Planner (unfortunately my GCS operator was struggling with sun on the screen and didn’t see them!).

I switched to Loiter mode, with the intention of manually flying back to land. However after a few seconds of hovering while I yawed slowly toward a convenient orientation, the copter began uncommanded rolling to the left (the artificial horizon did not correctly reflect this), I tried changing flight modes with no effect, as it had then gone into EKF failsafe and rolled right over, accelerating down and into the sea. The motors were going near full throttle, as the aircraft was trying to maintain height, thinking that it was the other way up! I could not regain control and hit the emergency motor cut switch before impact.

The roll indicated was way off at the end of the flight. The actual roll angle was more than 90 degrees to the left, and the artificial horizon was showing slight roll right!

Occasional bad gyro health errors were seen on a small number of previous flights and there had been some discussion of upgrading the PixHawk to to a cube. However, as it’s an off the shelf platform the wiring would have been a bit tricky.

My suspicions are:
-Gyro 2 producing dodgy data. I notice that it drifted in the Z axis quite a lot on the ground.
-Perhaps the compass calibration, or another problem with the compass. There were yaw inconsistent errors even though the indicated yaw was accurate. (This was the reason for the restarts visible in the log before flight.) The environment was not ideal for compass calibration (there could have been metal in the ground perhaps), although the autopilot seemed happy with the calibration we did. The aircraft had flown several times without giving errors (as far as I’m aware) between that calibration and this flight.

If a bad gyro was the culprit, I’m not sure why the EKF went with the bad gyro and not the good one. Perhaps because the compass data was off? :frowning:

I noticed tat he EKF2 and EKF3 produced quite different attitude estimates at various points in the flight. Not sure why that would be.

That link has an access requirement.

Apologies, hopefully that’s sorted now.

Hi David,
I had a quick look at the file and it appears drone had sudden vibrations on z axis when problems started.
Looks very similar what I had years ago and as a consequence lost an octocopter. Eventually found it again after 10 months. Needless to say wasn’t of much use anymore.
The likely cause was I had used plastic propellers at that time and would appear the “clever manufacturer” didn’t UV stabilize the plastic. As a consequence after some flights and obviously time spent in the sunlight the props became brittle. - Which I didn’t notice.
So on that particular flight a prop suddenly snapped but didn’t come fully apart and hence was flopping around to such extend that FC sensors couldn’t deal with it and I lost all control over drone. Had to watch on as drone was flying away.
It appears you may have suffered something similar as vibrations got worse towards the end of the flight.
(Since then been mostly using carbon fibre props wherever possible.)

Thanks for looking Karl. I did notice the brief spike in Y vibrations (up to 15) but it seems settle very quickly. The props in the case were high quality T-motor carbon fibre and had been checked before flight. There was slightly more bearing play in one motor than the rest, but nothing excessive and had been flying like this for some time with no issues.

The vibes are all within what I understand to be acceptable limits (they stay under 25) and are about as high as one would expect for a large coaxial copter I think?

I don’t think it’s unusual to see a spike in vibrations during the “brake” manouver going from auto to loiter mode? The air flows are changing rapidly.

There are intermittent EKF amber warning and “bad gyro health” warnings much earlier in the flight, so I imagine that the problem wasn’t just something that happened immediately before the loss of control?
Edit: just realised, you said Z axis, will look at this again…

I agree with Karl @Karl_Schoelpple. It looks like there was an event that spiked the Z-axis vibration along with clipping event’s. FC came loose, broken prop, etc?

Not that it matters now but looking at your parameters you have some very low Rate filters configured. 10Hz is a typical minimum even for big craft. I also see you didn’t have the Dynamic Notch Filter enabled.

So I’ve looked again, the Z vibrations seem mainly to depend on flight direction relative to wind. I don’t see any big changes. What time are you seeing the spike? I see a spike on Y at 10:52:08?

The EKF amber warning and bad gyro health started at 10:50:27, before any clipping occurs, and before the octo passed within about 100m of me, I perhaps would have noticed any strange noises from the aircraft at that distance, and also any broken props. Unfortunately I was unaware of the EKF amber warning or gyro warning at that stage.

I note that the EKF warnings are worst when the speed is low, such as near waypoints. The loss of control occurred soon after coming to a hover.

Edit: I note that both clipping events occur during higher levels of Y-vibration at 10:50:45 and 10:52:08, not when the Z vibes are highest. Also, the wiki suggests that small levels of clipping (<100) are not a concern. The clipping reaches 2 on this flight.

Edit2: I notice that the first bad gyro health warning occurrs at 10:50:44, just before the Y-vibe spike and first clipping event. Perhaps they could be connected?

I’ve found another log from a previous flight. That shows similar levels of Z-vibration, and levels of x and Y which are similar to most of this log, but without the two spikes in Y.

I wouldn’t just focus on props. It could also be a loose cover, which would explain the change with flight direction. Also that motor you’ve mentioned or payload ?

On my Hexacopter discovered loose screws on the landing gear mounting plates an also on the battery mount. Given they all been checked in the past it is clear that somehow the worked themself loose over various flights. In fact had to replace two as they were completely missing. - Luckily discovered it before any flight related problems occurred.

In regards to vibrations limits: From my own experience I’ve learned the hard way that you can’t always rely on those given limits. It appears if you have a combination of vibrations at the same time then you can have a problem even below those given levels. - I at least I experienced that over time.

I agree.
It’s been generally accepted that the grey area for vibration limits is up around 30, yet more realistically it could be 15 to 30 = “this could go either way” and above 25 or 30 is definitely bad.
Drastic maneuvers can cause a big jump in vibrations, but that can be expected. It is the constant vibration level that is important to minimize and keep checking periodically.
As you found inspections are worthwhile, and probably some thread locker :slight_smile:

Any clipping reported is a bad sign, especially if vibration appears low. That means the frequency was beyond what could be reported in logs.