Crash cause analysis

[ljurjevic (Luka Jurjevic)]

Hello everybody!
Seems like I will need your help.
We got the new custom-made drone back in September. We did two test flights with it, it seemed to work fine, but on the second flight, it crashed. It crashed very unexpectedly. We were flying on the height of cca 5m AGL in the Loiter mode, and while there was no input from our side (it was in the same position for some time), it started swinging in the pitch direction viciously (DesPitch and Pitch graph below) and soon (a few seconds later) crashed. In my opinion, it looks like ESC failure.
The firmware used is v3.5.7.
Copter type is OctaQuad X frame
Autopilot is the Cube

I will provide the *.bin file in the attachment of the flight that ended up with the crash and one just few minutes before it when it did not crash.

Here is the auto analysis, but some of the checks are not representative because the cause of the errors happened after the crash (e.g. Test: Compass = FAIL - Large change in mag_field (42.77%)).

Log File C:\Users\Luka Jurjevic\AppData\Local\Temp\tmp6C21.tmp.log
Size (kb) 18951.228515625
No of lines 272453
Duration 0:02:36
Vehicletype ArduCopter
Firmware Version V3.5.7
Firmware Hash b11c6af3
Hardware Type
Free Mem 0
Skipped Lines 0
Test: Autotune = UNKNOWN - No ATUN log data
Test: Brownout = GOOD -
Test: Compass = FAIL - Large change in mag_field (42.77%)

Test: Dupe Log Data = GOOD -
Test: Empty = GOOD -
Test: Event/Failsafe = GOOD -
Test: GPS = GOOD -
Test: IMU Mismatch = UNKNOWN - No IMU log data
Test: Motor Balance = WARN - Motor channel averages = [1555, 1511, 1583, 1508, 1566, 1493, 1569, 1529]
Average motor output = 1539
Difference between min and max motor averages = 90
Test: NaNs = GOOD -
Test: OpticalFlow = FAIL - FAIL: no optical flow data

Test: Parameters = FAIL - ‘THR_MIN’ not found
Test: PM = FAIL - 13 slow loop lines found, max 24.82% on line 107496
Test: Pitch/Roll = UNKNOWN - ‘BarAlt’
Test: Thrust = GOOD -
Test: VCC = GOOD -

So we blamed on the PID settings, but even after fixing the drone (which was quite expensive), and setting up the PID settings I am still not sure the issue is fixed. Also worth to note - PIDs are tunned with the dummy weight, while test flights are performed without any weight (camera payload).
I would appreciate if somebody checks the ATT log data, I am still pretty new to all this and I can not pinpoint the errors.

Here are the graphs of the accelerometer data and vibration graphs after fixing it and doing the test flight.
Acc data seems to be way out of the recommended values (±3 for x and y, and -10±5 for z).

Auto analysis:

Log File C:\Users\Luka Jurjevic\AppData\Local\Temp\tmp3CBD.tmp.log
Size (kb) 98867.263671875
No of lines 1443988
Duration 0:13:57
Vehicletype ArduCopter
Firmware Version V3.5.7
Firmware Hash b11c6af3
Hardware Type
Free Mem 0
Skipped Lines 0
Test: Autotune = UNKNOWN - No ATUN log data
Test: Brownout = GOOD -
Test: Compass = GOOD - mag_field interference within limits (1.79%)

Test: Dupe Log Data = GOOD -
Test: Empty = GOOD -
Test: Event/Failsafe = GOOD -
Test: GPS = GOOD -
Test: IMU Mismatch = UNKNOWN - No IMU log data
Test: Motor Balance = GOOD - Motor channel averages = [1646, 1610, 1586, 1606, 1619, 1638, 1615, 1576]
Average motor output = 1612
Difference between min and max motor averages = 70
Test: NaNs = GOOD -
Test: OpticalFlow = FAIL - FAIL: no optical flow data

Test: Parameters = FAIL - ‘THR_MIN’ not found
Test: PM = FAIL - 72 slow loop lines found, max 26.17% on line 1284280
Test: Pitch/Roll = GOOD -
Test: Thrust = GOOD -
Test: VCC = GOOD -

LOGS: https://www.dropbox.com/sh/hs4u03ec0invkrj/AACgSkMVP2OJj4mVLhG2rsz9a?dl=0

Any help will be greatly appreciated!
Thanks!

[mlebret]

@ljurjevic

Any help will be greatly appreciated!

Hello,

Not easy, but some clues…

Can you confirm, weight, size, motors and propellers.

There is something strange (like some motor are misplaced or not, mission planner log graph is better) when I look RC_OUT log. Can you check that each esc-motor group is well connected to the right RC Out plug.

On the latest log (after repair) there is a problem with compass switching during all flight.

Marc

[ljurjevic (Luka Jurjevic)]

Hello! Thank you for the reply!

This is the drone itself https://www.dropbox.com/s/kyve54iiwuco5zo/IMG_20180828_101558.jpg?dl=0

With the full payload it has 9kg, but during our flights it did not have a camera and gimbal, so I would say it is around 7kg.

Do you mean that they are physically displaced? Wouldnt that result in the crash soon after takeoff?
I do not see the same problem in the log after the repairing, RCOUT seems to be ok there.

I can’t confirm that they are all well connected since I did not do it, but I will be able to confirm it on Monday.

I will check the compass issue, at the moment I do not know what could have caused that.

[mlebret]

@ljurjevic

With the full payload it has 9kg, but during our flights it did not have a camera and gimbal, so I would say it is around 7kg.

Hello,

I will compare your settings with mine (OctoQuadX weight 5.4kg 15" props).

ATC_ACCEL_Y_MAX	27000
ATC_RATE_FF_ENAB	1
ATC_ACCEL_R_MAX	110000
ATC_ACCEL_P_MAX	110000

These values are quite hight. I have 9000 Yaw and 48000 Roll and Pitch

ATC_RAT_RLL_P	0.30000001192092896
ATC_RAT_RLL_I	0.18000000715255737
ATC_RAT_RLL_D	0.00800000037997961
ATC_RAT_RLL_IMAX	0.4000000059604645

ATC RAT RLL/PIT I : I have 0.1 (work in progress)

ATC RAT RLL/PI D : I slowed it to 0.005 as I had vibrations before take off. It is also a work in progress,

Your crash was due to an uncontrolled pitch and roll oscillation. Reducing I and D could help to avoid the overshoot.

Marc

[ljurjevic (Luka Jurjevic)]

@mlebret

Your crash was due to an uncontrolled pitch and roll oscillation. Reducing I and D could help to avoid the overshoot.

Marc

Can you comment on the DesPitch/Roll/Yaw and Pitch/Roll/Yaw from the log after repairing the copter?
Do you think that PID settings are ok now?

Thanks!

[mlebret]

@ljurjevic

Can you comment on the DesPitch/Roll/Yaw and Pitch/Roll/Yaw from the log after repairing the copter?
Do you think that PID settings are ok now?

Hello,

Looking at your log before and after crash, there is no change in parameters

So the problem could happen again.

ATC_ACCEL_Y_MAX	27000
ATC_RATE_FF_ENAB	1
ATC_ACCEL_R_MAX	110000
ATC_ACCEL_P_MAX	110000
ATC_ANGLE_BOOST	1
ATC_ANG_RLL_P	6
ATC_ANG_PIT_P	6
ATC_ANG_YAW_P	5
ATC_ANG_LIM_TC	1
ATC_RAT_RLL_P	0.30000001192092896
ATC_RAT_RLL_I	0.18000000715255737
ATC_RAT_RLL_D	0.00800000037997961
ATC_RAT_RLL_IMAX	0.4000000059604645
ATC_RAT_RLL_FILT	20
ATC_RAT_RLL_FF	0
ATC_RAT_PIT_P	0.30000001192092896
ATC_RAT_PIT_I	0.18000000715255737
ATC_RAT_PIT_D	0.00800000037997961
ATC_RAT_PIT_IMAX	0.4000000059604645
ATC_RAT_PIT_FILT	20

Look at http://ardupilot.org/copter/docs/parameters.html#atc-parameters

and reduce your ATC_ACCEL_*_MAX at VerySlow on Yaw, Roll and Pitch

You could also reduce ATC_RAT_PIT_I and ATC_RAT_RLL_I around 0.15 or lower.

Lower the ATC_RAT_PIT_ and RLL_FILT at 10

Fly with it and look at the result.

Marc

[ljurjevic (Luka Jurjevic)]

@mlebret

Looking at your log before and after crash, there is no change in parameters

Marc

Hello Marc!

Are you sure there is no change in the parameters?

I read this out of the log of the crash flight (before) and the one after repair (after):

Parameter			Before	After

ATC_SLEW_YAW 		6000	6000
ATC_ACCEL_Y_MAX 	27000   27000
ATC_RATE_FF_ENAB 	1       1
ATC_ACCEL_R_MAX 	110000  110000
ATC_ACCEL_P_MAX 	110000  110000
ATC_ANGLE_BOOST 	1       1
**ATC_ANG_RLL_P 	8       6**
**ATC_ANG_PIT_P 	8       6**
ATC_ANG_YAW_P 		5       5
ATC_ANG_LIM_TC 	    1       1
ATC_RAT_RLL_P 		0.3     0.3
ATC_RAT_RLL_I 		0.18    0.18
**ATC_RAT_RLL_D 	0.004   0.008**
ATC_RAT_RLL_IMAX 	0.4     0.4
ATC_RAT_RLL_FILT 	20      20
ATC_RAT_RLL_FF 	    0       0
ATC_RAT_PIT_P 		0.3     0.3
ATC_RAT_PIT_I 		0.18    0.18
**ATC_RAT_PIT_D 	0.004   0.008**
ATC_RAT_PIT_IMAX 	0.4     0.4
ATC_RAT_PIT_FILT 	20      20
ATC_RAT_PIT_FF 	    0       0
ATC_RAT_YAW_P 		0.25    0.25
ATC_RAT_YAW_I 		0.05    0.05
ATC_RAT_YAW_D 		0       0
ATC_RAT_YAW_IMAX 	0.25    0.25
ATC_RAT_YAW_FILT 	2.5     2.5
ATC_RAT_YAW_FF 	    0       0
ATC_THR_MIX_MIN 	0.1     0.1
ATC_THR_MIX_MAX 	0.5     0.5
ATC_THR_MIX_MAN 	0.5     0.5

Aren’t this parameters the ones that have the highest influence on the flight performance?

[mlebret]

@ljurjevic

Are you sure there is no change in the parameters?

I read this out of the log of the crash flight (before) and the one after repair

I missed the changes.

You can lower ATC_ACCEL_Y_MAX to 9000
ATC_ACCEL_R&P_MAX to 48000

ATC_RAT_ROLL&PIT_I to 0.15 or lower

Pay attention at ATC_RAT_RLL&PIT_D. If you have small and rapid motor vibrations when reaching flight power, you can lower it. Too large D is not good.

Marc

[ljurjevic (Luka Jurjevic)]

@mlebret

@ljurjevic

Are you sure there is no change in the parameters?

I read this out of the log of the crash flight (before) and the one after repair

I missed the changes.

You can lower ATC_ACCEL_Y_MAX to 9000
ATC_ACCEL_R&P_MAX to 48000

ATC_RAT_ROLL&PIT_I to 0.15 or lower

Pay attention at ATC_RAT_RLL&PIT_D. If you have small and rapid motor vibrations when reaching flight power, you can lower it. Too large D is not good.

Marc

Thanks Marc!
I will try it out!

[mboland (Mike Boland)]

For a start looking at the 13-59-41.bin you seem to have a real vibration problem so you are correct, may above expected maximums.
I am more curious why it doesn’t show up in the Vibe log but the IMU are way too high and Vibe clipping started when things went south.

The repeated switching between compasses would indicate a compass issue.
Have you done a motor/compass calibration?
I noted that in the crash log you didn’t have the compass switching.

It was the pitch that started to oscillate out of control, although the roll was not good, just not out of control before the pitch.
The frame is not overpowered, running at about 1600 to 1700 PWM values.
If this is without payload it is going to be struggling with another 2kg.
Could it be that it has been tuned to be running nearly flat out?
Have you tried it with a dummy 2kg?

You are going to have to sort out those vibrations before anything else.

[ljurjevic (Luka Jurjevic)]

@mboland

For a start looking at the 13-59-41.bin you seem to have a real vibration problem so you are correct, may above expected maximums.
I am more curious why it doesn’t show up in the Vibe log but the IMU are way too high and Vibe clipping started when things went south.

The repeated switching between compasses would indicate a compass issue.
Have you done a motor/compass calibration?
I noted that in the crash log you didn’t have the compass switching.

It was the pitch that started to oscillate out of control, although the roll was not good, just not out of control before the pitch.
The frame is not overpowered, running at about 1600 to 1700 PWM values.
If this is without payload it is going to be struggling with another 2kg.
Could it be that it has been tuned to be running nearly flat out?
Have you tried it with a dummy 2kg?

You are going to have to sort out those vibrations before anything else.

This vibration issue is giving me a headache…
The Mean values of the AccXYZ data are just as they should be - 0,0,-10. However, there is so much noise there. Still, VIBE and CLIP data is in the acceptable range…
In a way it means that my IMU readings are pretty consistent, would “INS_ACCEL_FILTER” parameter change anything? It is set to 20hz at the moment.

Yes, and the compass is also pretty consistent. No idea why would it switch all the time. I will assume it has something to do with the GPS sensor. We have 2 GPS sensors (here and here+), however, I noticed that *.bin file contains only data from one sensor. That seems very odd.

And PIDs were tuned having the dummy weight under the copter. It really does not seem to struggle when flying. It seems to have a lot of power, but I will deal with this problem later. Vibrations are now my main concern… Can’t let the drone go on the auto mission if I am not sure it is stable enough and won’t get the EKF fail… That would be catastrophic as it would attempt to land above very dense forest!

[ljurjevic (Luka Jurjevic)]

@mlebret

@ljurjevic

Are you sure there is no change in the parameters?

I read this out of the log of the crash flight (before) and the one after repair

I missed the changes.

You can lower ATC_ACCEL_Y_MAX to 9000
ATC_ACCEL_R&P_MAX to 48000

ATC_RAT_ROLL&PIT_I to 0.15 or lower

Pay attention at ATC_RAT_RLL&PIT_D. If you have small and rapid motor vibrations when reaching flight power, you can lower it. Too large D is not good.

Marc

I did the next changes: was / is now
ATC_ACCEL_Y_MAX 27000 9000
ATC_ACCEL_R_MAX 110000 48000
ATC_ACCEL_P_MAX 110000 48000
ATC_RAT_RLL_FILT 20 10
ATC_RAT_PIT_FILT 20 10

• log bitmask change

I tested the parameters and concluded two things.

1. Vibrations are more than OK, they are great.
2. Applying ATC_ parameters improved the movement of my copter. It flies really smooth now, maybe additional tuning is necessary but I will do it via autotune if I do it.

The thing with the vibrations is that enabling all IMU data logging (IMU, IMU_FAST and IMU_RAW) apparently saves only raw data? When I disabled IMU_RAW and IMU_FAST and enabled IMU only, I got a really nice accx, accy and accz data, apparently the data that should be checked for this test is not ACC1.AccX but IMU.AccX. The wiki is out of date it seems and it made me circle in the spot.

Here is the log if somebody wants to check out - Dropbox - File Deleted - Simplify your life

Log File C:\Users\Luka Jurjevic\AppData\Local\Temp\tmpCD77.tmp.log
Size (kb) 28482.40625
No of lines 265771
Duration 0:00:00
Vehicletype ArduCopter
Firmware Version V3.5.7
Firmware Hash b11c6af3
Hardware Type
Free Mem 0
Skipped Lines 0
Test: Autotune = UNKNOWN - No ATUN log data
Test: Brownout = GOOD -
Test: Compass = GOOD - mag_field interference within limits (2.27%)
Test: Dupe Log Data = UNKNOWN - No ATT log data
Test: Empty = GOOD -
Test: Event/Failsafe = GOOD -
Test: GPS = GOOD -
Test: IMU Mismatch = GOOD - (Mismatch: 0.20, WARN: 0.75, FAIL: 1.50)
Test: Motor Balance = GOOD - Motor channel averages = [1564, 1579, 1542, 1532, 1577, 1566, 1530, 1543]
Average motor output = 1554
Difference between min and max motor averages = 49
Test: NaNs = GOOD -
Test: OpticalFlow = FAIL - FAIL: no optical flow data
Test: Parameters = FAIL - ‘THR_MIN’ not found
Test: PM = FAIL - 28 slow loop lines found, max 13.88% on line 165295
Test: Pitch/Roll = UNKNOWN - No ATT log data
Test: Thrust = UNKNOWN - No ATT log data
Test: VCC = GOOD -

Applying ATC_ parameters improved the movement of the copter really a lot. I did not log the ATT data so I can not attach the ATT graph, but I will log it next time.
Having log bitmask that logs “ALL” data seems to be very costly in terms of computational power. Before I used to have cca 20% - 25% of the slow lines in the PM data, however, after disabling some data it dropped down to 13%. That seems to have some influence on flight stability.

The thing that is bothering me now is next - I have only GPS2 data in the log! However, there were two GPS devices connected and I could read data from both of them (Hdop, Hdop2 etc)
Here is a quote from the LOG file:

MSG, 166912562, GPS 1: detected as u-blox at 115200 baud
MSG, 166912599, u-blox 1 HW: 00080000 SW: EXT CORE 3.01 (db0c89)
MSG, 166912636, GPS 2: detected as u-blox at 115200 baud
MSG, 166912673, u-blox 2 HW: 00080000 SW: EXT CORE 3.01 (107900)

Anybody knows what is the issue here? Cant find anyone that had such a problem.
Also - sometimes it logs both of them, I really can’t find the pattern of such behaviour.

Any help is much appreciated!
Luka

[mlebret]

@ljurjevic

The thing that is bothering me now is next - I have only GPS2 data in the log! However, there were two GPS devices connected and I could read data from both of them (Hdop, Hdop2 etc)

Hello,

Happy to see some progress.

Relevant Vibration data for your config is Vibration 3.3 (also good in the log).

One GPS data missing from the log?? No clue, Msg are consistent with good GPS detection by the board.

Some other parameters are missing, like Attitude data relevant to diagnose PID settings.

Can you change your LOG_BITMASK value to 63485 or 63486.

GPS settings: GPS_AUTO_CONFIG is set to 1 (enable automatic configuration) but GPS_GNSS_MODE (1 and 2) is set to 0 (leave as currently configured). If one GPS config is bad, it could be the cause of missing data. You can force GPS_GNSS_MODE to 67 for both GPS (GPS+GLONASS+SBAS) and report result.

At home without flying or arming, just with powering the board, you can verify GPS data in Status screen.

Marc

[Lawrence_Dennis]

Have you thought about rotor vortex ringstate? What is the Voltage of the aircraft, What is the motors KV and size , and Prop size? (length & Pitch)

I have seen this behavior with coaxial frames and short boom diameters or low KV motor prop combinations. It may not be the PID’s or Autopilot… Just thinking outside the box…

[ljurjevic (Luka Jurjevic)]

@mlebret

@ljurjevic

The thing that is bothering me now is next - I have only GPS2 data in the log! However, there were two GPS devices connected and I could read data from both of them (Hdop, Hdop2 etc)

One GPS data missing from the log?? No clue, Msg are consistent with good GPS detection by the board.

GPS settings: GPS_AUTO_CONFIG is set to 1 (enable automatic configuration) but GPS_GNSS_MODE (1 and 2) is set to 0 (leave as currently configured). If one GPS config is bad, it could be the cause of missing data. You can force GPS_GNSS_MODE to 67 for both GPS (GPS+GLONASS+SBAS) and report result.

At home without flying or arming, just with powering the board, you can verify GPS data in Status screen.

Marc

Hello Marc thanks for the help!

Yes I do see GPS status and I can read all the data live while the copter is connected to the GCS, It seems to work like a charm.
However, when I arm the Copter it starts logging data and only data from one GPS is in the log.
I have HERE and HERE+ GPS devices for redundancy. I see that they both are working and give a similar solution, but I sometimes get the data from one device logged and there is no logged data from the other one.

I do not understand the parameter GPS_AUTO_CONFIG fully. Does it bypass the other parameters set for GPS? Does it take into account e.g. GPS_GNSS_MODE parameter that I set if GPS_AUTO_CONFIG is 1.

These re the parameters that I changed, but only after changing the LOG_BITMASK it started logging GPS and GPS2 data. Quite unexpected. Still I will confirm on Monday did it really solve the issue. I did notice that sometimes it logs data from both gps and gps2, but quite rarely.

GPS_GNSS_MODE		0	67
GPS_GNSS_MODE2		0	67
LOG_BITMASK		194300	63486

[ljurjevic (Luka Jurjevic)]

@Lawrence_Dennis

Have you thought about rotor vortex ringstate? What is the Voltage of the aircraft, What is the motors KV and size , and Prop size? (length & Pitch)
I have seen this behavior with coaxial frames and short boom diameters or low KV motor prop combinations. It may not be the PID’s or Autopilot… Just thinking outside the box…

Wow, I really hope that is not the problem.

The voltage of the aircraft? There are two batteries and log says it is cca 24.5V fully charged.

It is equiped with 8 T-Motor Antigravity 4006 KV380 - MN4006 Antigravity Type 4-6S UAV Motor kv380 - 2PCS/SET_Antigravity Type_Motors_UAV Power_T-MOTOR Official Store - UAV Power System, Robot Power System, Model Power System
Props are T-Motor P16x5.4 Prop - http://store-en.tmotor.com/goods.php?id=382

Also, could you explain what is boom diameter? The distance between diagonal motors? I would say it is quite short, 71cm.

I would appreciate your comment on this!
Thanks!

[Lawrence_Dennis]

Hello Luka,

So it looks like your a 6S copter with 380 KV engines and 16" props, this coax setup is on the edge of rotor vortex ring state. The Diameter I spoke of is motor to motor dimensions, meaning if the arms are short then we have a faster rolling moment on the aircraft. You may want to try going to 17 inch props and see if it fixes the issue, or up the KV to over 400. There are kind of two problems here, Frame engine to engine diameter or rotor ring state.

Rotor ring state will effect every flight mode you have. 3 things need to happen to get into rotor ring state, 1: aircraft descending (falling into its own rotor wash) 2: directional speed of less than 3 Kph (basically in a hover or having wind keep the rotor wash with the aircraft. 3: engines are delivering over 20% power.

If those three things all happen at the same time then the propeller on the engine will stall even though it is spinning at the speed it should.

Settling with power (rotor ring state) becomes more prevalent on aircraft under load, or larger aircraft. The smaller light aircraft with faster spinning motors usually avoid these situations.

The motor to motor diameter cold also be an issue, you have strong engines with a short arm length then you can get out of the range of what the PID’s can handle.

[ljurjevic (Luka Jurjevic)]

Hello Lawrence,

Thank you for the comment, I did some reading and watching today. I assume you are 100% right that this setup has some potential to get into the rotor ring state - I prefer the copters with larger arms, definitely.

However, I am pretty sure that this specific crash did not happen due the rotor ring state.
From what I have seen, rotor ring state causes wobble and more or less regular descent and ground impact due to propellers stalling.
In this specific case I am pretty sure propellers did not stall. The copter had enough thrust and fell due to large angles of the oscillations. While transitioning from one maximum to the other the copter did not lose height while in the horizontal attitude state, but only when the tilt was almost vertical. That is why I think it did not cause the crash.

Nevertheless, this is a big and scary issue and problem that should not be ignored. And dealing with it includes some large changes in my usual flight planing!

[ljurjevic (Luka Jurjevic)]

Here is my log from the recent flight.
The log has ATT message logged. I would really appreciate if somebody commented on the log. More precisely, on the ATT message, Des values and true values.

link for the log - https://www.dropbox.com/s/tx3dbcn61skhfn1/2019-02-07%2011-33-55.bin?dl=0

I am not sure is this OK or should I do the autotune? The producer of the copter is not recommending the autotune, saying that he had some bad experience with it.

Any help is much appreciated guys!

[mboland (Mike Boland)]

I don’t see any real crash as such.
Your ATT messages are not too bad, and follow well until the end of the flight where you dropped the throttle while it still appears to be in the air.

As you were in Stab mode the motors stopped

A crash is not recognised in the log just a Land.