Crash of Octocopter, crash check released in mid-air?

[Julian]

Hi everyone!

I use ArduCopter for around 4 years and have read in this forum many times for various reasons. This is the first time writing because I need your help with the analysis of a crash, we had some time ago. Our commercially used Octocopter, worth around 12 k Euros (13 k$), fell from the sky like a stone from a height of about 80 m. Fortunately, there have not been any personal Injuries on the industrial site we were flying. We would be happy if you could help us understanding the exact course of events in order to reduce the risk for future flights.

Vehicletype ArduCopter
Firmware Version V4.0.1-rc2
Firmware Hash f6121406
Logfile: https://easyupload.io/7io8t5
BinFile: https://easyupload.io/042t49

We already did some research with the log file and came up with a basic theory:
• The copter is following some waypoints in Auto Mode
• The ESC or motor No. 8 suddenly fails. The copter loses some height but does not fall down since the other motors are able to keep the copter in the air. There is a thrust loss error shortly after the motor fails.:

• The copter drifts/flies away and loses even more height and cannot be navigated back to the original flying zone by the pilot. The pilot switches to Loiter mode and later to altHold mode and then even RTL without any results.
• After some time, the mode is switched back to Auto (by the pilot we suppose) and this tells the copter to fly to the next waypoint, which (for some reason) works and the copter is flying back to position near the pilots and even regains its original height.

• The pilot plans to land the copter, it starts rotating and suddenly all motors stop and the copter falls down. The log file suggests that a crash check has been released an stopped the motors.
Of course, a failure of an ESC is something that can happen anytime. However, we thought that an Octocopter should still be able to keep on flying for a safe landing. And apparently before the crash this was the case here.

We still have the following Questions.

1. Can you verify or disprove our theory of the events?
2. Why did the coper drift/fly away after the motor/ESC failure? And why did it come back when in Auto Mode?
3. Why did the copter stop the motors in mid-air? Is the crash check function supposed to work this way or is there a need to improve the software somehow?

Thanks in advance for your help!

[smartdave (Dave Glover)]

Can you upload the .bin file? The .log doesn’t really help

[Julian]

I added a link to the .bin file in the first post.

[Julian]

Is there anyone with an opinion on the topic? We would appreciate any advice you have!

[mboland (Mike Boland)]

Looks like you lost Motor 8, at least intermittently
Initially you got an error SubSys:25

which is where it initially started losing altitude

So you can see here where it was fighting the power loss

The curious part is the final motor cut which is highlighted by the Mission: 1 Takeoff command being executed.

Looking at the power usage there is a lot of current spikes when M8 failed.
Short?

The only other anomaly in the log I could find was an EKF event just before the midair disarm.

But there was no EKF errors in the log and I am not familiar enough with the EKF nuances to say it was a cause.
A midair disarm is not good so it would be nice if @tridge or @rmackay9 or other EKF experts had a look at this.

[xfacta (Shawn)]

Being a Cube, definitely upgrade to 4.0.3 as soon as possible.
Your X axis and Z axis vibrations are almost always in a “grey area” and you should probably closely examine this. There is clipping during the entire flight.
I also couldn’t see the exact reason for midair disarm and can only assume it’s a combination of specific conditions around loss of thrust and loss of attitude control.

Before issues with the motor, there’s a physical yaw imbalance for the whole flight, CCW motors are working harder to fight CW motors.This by itself is not critical, but once you lose a motor the flight controller has 2 problems to fight against, and it loses yaw control by trying to maintain attitude.
In this case you almost lose pitch and roll too, there’s definitely oscillations there. You can see that once one motor is lost some other motors are nearly at maximum to maintain flight.

Once the motor fails and Loiter then AltHold is selected, there’s almost no throttle down or attempts to lose altitude (the good way) and land. I’m not sure if surroundings prevented a landing, but I’d have recommended immediate landing even in Stabilise mode. I’d be using any redundancy an Octocopter provides to get landed ASAP. Sorry to lecture or point fingers… A pilots natural tendency is probably to try and keep flying and even complete the mission - not always the best outcome.

I took a guess at your prop size; To improve stability I’d recommend setting these and running Autotune again after you’ve fixed the crash damage and yaw bias.
ATC_RAT_PIT_FLTD,15
ATC_RAT_PIT_FLTT,15
ATC_RAT_RLL_FLTD,15
ATC_RAT_RLL_FLTT,15
ATC_RAT_YAW_FLTE,2
ATC_RAT_YAW_FLTT,15
ATC_THR_MIX_MAN,0.5
MOT_THST_EXPO,0.71
PSC_ACCZ_I,0.6
PSC_ACCZ_P,0.3

You craft might also benefit a lot from the Dynamic Harmonic Notch Filter
https://ardupilot.org/copter/docs/common-imu-notch-filtering.html

And set these too:
BATT_CRT_VOLT,21
BATT_FS_CRT_ACT,1

Use this spreadsheet to help with initial parameters

Initial parameters.xls

It’s based on this Tuning Guide:
https://ardupilot.org/copter/docs/tuning-process-instructions.html
And now it’s implemented in MissionPlanner too!
https://discuss.ardupilot.org/t/initial-parameters-calculator-plugin

EDIT: I did some graphs of your data, but they got messy as I added more and more data

[Julian]

Thanks for the replies so far!

@xfacta

Being a Cube, definitely upgrade to 4.0.3 as soon as possible.

Is this a general advice or do you think of any specific fix we might benefit from?

@xfacta

Once the motor fails and Loiter then AltHold is selected, there’s almost no throttle down or attempts to lose altitude (the good way) and land. I’m not sure if surroundings prevented a landing, but I’d have recommended immediate landing even in Stabilise mode. I’d be using any redundancy an Octocopter provides to get landed ASAP. Sorry to lecture or point fingers… A pilots natural tendency is probably to try and keep flying and even complete the mission - not always the best outcome.

I have not been on the site during the flight. One reason for not landing was, that pitch and roll could not be controlled after the motor failure and the copter was drifting away in an uncontrolled way to a location where the landing zone was not safe.
But still I agree with you and maybe an immediate reaction of the pilot might have prevented the situation.

@xfacta

I took a guess at your prop size; To improve stability I’d recommend setting these and running Autotune again after you’ve fixed the crash damage and yaw bias.

Yes, we will. We will also check the Dynamic Harmonic Notch Filter but I have to look into it more in detail to understand what it is.

From your answers I take the following hints, which we will follow up in the future:

• Update to 4.0.3
• Check Vibrations and check for clipping
• Check Yaw imbalance
• Autotune with correct initial parameters
• Train pilots to react properly in such a situations

It might be possible, that with those improvements, we will not experience such a failure again. However, we find ourselves in a position, where we have to make a recommendation to the operator of the plant and the drone for future flights. We would feel safer if we could understand the exact circumstances of the mid-air disarm.
Thank you anyway for your suggestions so far!

[mboland (Mike Boland)]

I would add one point here about your operators and their training.
When things go south the fallback mode should be Stabilise, full manual control.
The less you are relying on guided modes the more chance you have of regaining control.

You’re operators should instinctively revert to Stabilise.
Have been in this situation at a power station where a wandering copter could have disastrous consequences.

[ThePara (Cornel Fudulu)]

Stabilise is hard to train from start. Alt-hold is the no-GPS flight mode of choice.
If the operator doesn’t get enthusiastic about his job, even after 6 months and maybe 100 flights, Stabilise is still hard to get used to. With two exceptions, and one was the only female operator I’ve trained, that’s my experience so far.

[VDLJu (LJu)]

I want to say that @ThePara is not alone with that opinion. Pretty much all professional operators don’t have the time or enthusiasm to learn to fly copters in stabilize mode. They want to handle their main business and use tools (drones) that you can fly in easy to fly modes.

Also note that flying in stabilize mode would need to be trained often, because in stressing situation you need to be quick and skillful. Otherwise the result can be an even less controlled crash.

Sadly, I would say that for many even AltHold is too much to bear with.

[Corrado_Steri]

Than be prepared to loose your uav in an event something happens. I, for sure, wouldn’t like the option of having an Hasselblad or a Red carried by not skilled pilot

[Julian]

I would like to put the focus back on the cause of the crash. I noticed another detail in the log data. The moment motor 8 fails, there starts a strong vibration in the IMU data. The vibrations continue while the copter is drifting away and seems to be uncontrollable until the copter stabilizes for a short time and flies back successfully.

In particular, you can see a strong amplitude in AccX and AccY. In my opinion this indicates an imbalance in one of the propellers. Maybe one side of a propeller (number 8) has broken off? Also the cyclical yaw of the copter (GyrZ) would indicate this.

The question is why the phenomenon stops after about one minute. After the crash, all propellers were still attached to the engines (at least partially, some are broken). So it is not possible that the defective propeller has come loose entirely.
Maybe the ESC firmware (BLHeli_S Revision 16.6) switched off the imbalanced engine because of the low rotation speed? The vibrations at AccX and AccY indicate a rotation speed of about 1Hz/60rpm.

Could such strong vibrations cause the copter to lose its “orientation” and not be controlled well anymore? What do you think?

EDIT: Why don’t my screenshots go fullscreen when I click on them like in other people’s posts?
EDIT2: Now they do. They had a bad resolution the first time.