CRASH! Help to understand. Copter 3.6.10 Cube Black

ArduCopter Copter 3.6
1

Hello,
Today I experienced an awful crash with my hexacopter. Unfortunately I have no video of the crash. The copter was flying fine in loiter then looks like it disarmed in air.

The logs: https://we.tl/t-nSTvp9tnCf

Hope you can help me to understand what happened.
Kind regards…

2

Here is the video of Mission Planner that shows a possible reboot in flight.

3

@samucs,

yes, from the video it does look like the flight controller rebooted. This doesn’t necessarily mean it was a software issue though of course. The most common cause is an issue with how the flight controller is being powered. Can you find the log after this one? It’s probably numbered 139 and if we can find that then it will have a line somewhere that tells us if it rebooted because of a software failure or not.

4

Hi @rmackay9

Here is the 139 log file: https://we.tl/t-o88h3KyeVg

Thanks for your help, I really wold like to understand what the problem is.

5

can you dig out the tlog from MissionPlanner? The tlog is likely to give us the most information

6

@samucs,

Thanks for the log, so we can confirm that this is a reboot in flight due to a software reboot because these messages appear in the logs

MSG, Restored watchdog attitude 1 16 196}

So this is very bad of course. We suspect this is caused by the I2C storm issue that is fixed in Copter-3.6.11 but it’s hard to be completely sure. As @tridge says above, if you’ve got a tlog that may tell us.

I’d highly recommend upgrading to Copter-3.6.11 if possible. It should be a smooth upgrade.

7

@rmackay9,

I’ve been reading some other posts that mentioned a watchdog rebooting the flight controller while in flight. It makes sense for airplanes, they glide. Copters on the other hand go straight down… is there any reason to have it for copters? Is it new code for 3.6? Finally is there a link that talks about the I2c storm issue you mentioned?

Will provide the tlog as soon I get it.
Kind regards…

8

@samucs, yes, this is new for 3.6.10 but to be clear, the flight controller was locking up one way or another. The change in behaviour is just that it restarted and now we know it was caused by some kind of software failure. Hopefully we can get to the bottom of this. Txs for the report.

1 Like
9

@rmackay9, @tridge,

Hello Tridge, here is the tlog: https://we.tl/t-wAcNMKWj8w
Hope it can tell us more information about this crash.
Still would like to know if this watchdog is really necessary for copters…

10

Yes, I think what Randy is saying is without the watchdog the FC would have locked up anyway but it wouldnt have rebooted. The idea is with the watchdog and reboot there’s at least an outside chance of recovery, or at minimum some evidence.

11

@tridge, were you able to have a look at the tlog?

12

When exactly did we see first crashes due to the I2C storm/ChibiOs bug ? It seems to me it was not an issue before 3.6.7 or so, yet 3.6.4 had the first fixes for I2C trouble.

It is time to write down history of this issue, and have some safe recommendations.

13

I recently crashed in air due to reboot of FC with 3.6.10 v on paixhawk 1. No one took time to look at my case thou but i am a victim as well of this issue.

14

can you give me a link to that report?

15

The first crash that we have subsequently attributed with some confidence to this issue was on April 15th, but it wasn’t until August 25th that the underlying cause was found. The issue is fixed in copter 3.6.11 and plane 3.9.11.
It is likely there were crashes before April due to this issue but we had no way to attribute them. It wasn’t until we had watchdog support that we could distinguish between a software bug and a hardware fault for this type of failure.
For the 4.0 releases we’ve added in a more generic method of preventing i2c interrupt storms of all types by limiting the number of interrupts allowed per transferred byte.

16

Here is the link to my crash report:

17

Is this issue specific to certain type of I2C bus use, or can it happen with any kind of usage?
How widespread this issue is?

2 Likes
18

I am also interested to know what would cause this issue. In another post, it was mentioned that the QMC5883L can trigger the interrupt storm; can it happen with other I2C device types?

Also, what are some of the symptoms in the dataflash logs that will hint at the occurrence of an I2C interrupt storm?

19

Hello, I got same crash with pixhawk 1 still disarm in air with 3.6.10 on hexacopter using QMC5883L…
I check the log after crash and I found the msg watchdog…

Question is upgrading to 3.6.11, Can I use safely this magnetometer? Or can you tell me witch magnetometer can be used to prevent i2c storm?

After crash i tested and i seen pixhawk reboot without being armed, (on this drone magnetometer cable 5v sda scl gnd is 40cm long).

I noticed that just seconds before reboot color led stop blinking or made not expected colors…

I upgraded to 3.6.11 and pixhawk never reboot again, But led color stop blink or made such not expected colors…

1 Like
20

3.6.11 is supposed to have i2c storm protection

You can see the release notes here (See 3f):

The way I read those notes, the fix applied in 3.6.11 does not appear to be make/model-specific