CRASH! Help to understand. Copter 3.6.10 Cube Black

samucs · October 8, 2019, 2:40pm

@tridge, were you able to have a look at the tlog?

Andre-K · October 8, 2019, 5:21pm

When exactly did we see first crashes due to the I2C storm/ChibiOs bug ? It seems to me it was not an issue before 3.6.7 or so, yet 3.6.4 had the first fixes for I2C trouble.

It is time to write down history of this issue, and have some safe recommendations.

MamboJambo · October 8, 2019, 9:01pm

I recently crashed in air due to reboot of FC with 3.6.10 v on paixhawk 1. No one took time to look at my case thou but i am a victim as well of this issue.

tridge · October 8, 2019, 9:50pm

can you give me a link to that report?

tridge · October 8, 2019, 9:56pm

The first crash that we have subsequently attributed with some confidence to this issue was on April 15th, but it wasn’t until August 25th that the underlying cause was found. The issue is fixed in copter 3.6.11 and plane 3.9.11.
It is likely there were crashes before April due to this issue but we had no way to attribute them. It wasn’t until we had watchdog support that we could distinguish between a software bug and a hardware fault for this type of failure.
For the 4.0 releases we’ve added in a more generic method of preventing i2c interrupt storms of all types by limiting the number of interrupts allowed per transferred byte.

MamboJambo · October 9, 2019, 4:42am

Here is the link to my crash report:

VDLJu · October 9, 2019, 6:48am

Is this issue specific to certain type of I2C bus use, or can it happen with any kind of usage?
How widespread this issue is?

disgruntled-patzer · October 13, 2019, 3:09am

I am also interested to know what would cause this issue. In another post, it was mentioned that the QMC5883L can trigger the interrupt storm; can it happen with other I2C device types?

Also, what are some of the symptoms in the dataflash logs that will hint at the occurrence of an I2C interrupt storm?

kuspower · October 17, 2019, 9:50pm

Hello, I got same crash with pixhawk 1 still disarm in air with 3.6.10 on hexacopter using QMC5883L…
I check the log after crash and I found the msg watchdog…

Question is upgrading to 3.6.11, Can I use safely this magnetometer? Or can you tell me witch magnetometer can be used to prevent i2c storm?

After crash i tested and i seen pixhawk reboot without being armed, (on this drone magnetometer cable 5v sda scl gnd is 40cm long).

I noticed that just seconds before reboot color led stop blinking or made not expected colors…

I upgraded to 3.6.11 and pixhawk never reboot again, But led color stop blink or made such not expected colors…

ekliptiko · October 17, 2019, 9:52pm

3.6.11 is supposed to have i2c storm protection

You can see the release notes here (See 3f):

github.com

ArduPilot/ardupilot/blob/master/ArduCopter/ReleaseNotes.txt

ArduPilot Copter Release Notes:
------------------------------------------------------------------
Copter 3.6.11 01-Oct-2019 / 3.6.11-rc1 16-Sep-2019
Changes from 3.6.10
1) EKF and IMU improvements:
    a) IMU3 enabled by default if present
    b) IMU3 fast sampling enabled by default on Cube autopilots
    c) EKF protection against large baro spikes causing attitude error
    d) EKF origin fixes (consistent across cores, set externally only when not using GPS)
    e) EKF logging of 3rd core
2) Minor enhancements:
    a) Land mode supports heading requests (ie. ROI)
    b) Support Hexa-H frame
    c) MatekF405-STD binaries created
    d) Benewake TFminiPlus lidar support
3) Bug Fixes
    a) Barometer health checks include sanity check of temperature
    b) Lightware serial driver handles invalid distances
    c) IO firmware fix involving delayed writes to serial ports (ChibiOS only)
    d) CAN Compass fis to for unintialised device IDs

This file has been truncated. show original

The way I read those notes, the fix applied in 3.6.11 does not appear to be make/model-specific

rmackay9 · October 18, 2019, 12:24am

@kuspower, really sorry about your crash. I hope the damage wasn’t too bad. Thanks very much for the report though. TBH, I’m slightly surprised at how many reports we are getting.

@disgruntled-patzer, I’ve asked @tridge if he has a better idea of which I2C devices can cause the problem. I haven’t personally seen the list.

So far we haven’t seen any repeats of the issue with Copter-3.6.11 users so we strongly recommend users upgrade to 3.6.11 if they’re using something older. Perhaps we need to make a more visible annoucement of this recommendation.

kuspower · October 18, 2019, 10:07am

Thank you Randy, No worry.

Yes I think is good idea to really force user to upgrade to 3.6.11.

Also, if there is list of sensors that can cause troubles, it can be interesting to share it.

Idea is to secure users.

Thank you so much.

samucs · October 18, 2019, 8:55pm

I would like to report that after fixing the airframe, replacing propellers, and updating the firmware to 3.6.11, the drone is flying again with no issues. It’s doing auto missions fine. GPS modes, rangefinders, grippers and precision landing are working as expected. Hopefully this is really a stable version, we will continue to test fly in the following days. Thanks!

ThePara · October 20, 2019, 8:37am

I’ve changed the GPS units on two identical hexacopters, sporting Pixhawk 1s with 3.6.9, and since the replacement M8Ns lacked mags, I’ve used QMC5883s from a local Arduino shop.
One kept rebooting on the ground, the second started throwing compass errors after takeoff, so I landed immediately. Replacement mags - original HMC5983 - arrived yesterday, and I’ll start testing monday.

Houss_Ribeiro · October 20, 2019, 9:38pm

Hi there i have a 3DR Iris flying smoothly the only think by yesterday the 2D tarot gimbal stops tilt usingvthe knob at the transmitter any idea why and how to fixe it?
Regards
Houss

samucs · October 21, 2019, 11:06am

@Houss_Ribeiro Hi, welcome to the community. I suggest you start a new thread and ask for help on this particular issue. This thread is about solving a crash that myself and other people experienced with Copter 3.6.10. It is recommended that all users upgrade to 3.6.11 to avoid the I2C storm problem that seems to be the root cause for our crashes.

Houss_Ribeiro · October 21, 2019, 12:04pm

Hi Samuel. Sorry for using a wrong space to address my issue. Thx once again to welcome myself in the community. Anyways i did solved my issue.
Have a great day

Houss Ribeiro

MamboJambo · October 23, 2019, 3:14pm

Hello, I just had carbon copy crash on P’xhawk 1 Nuttx 3.6.7 version as i have been flying with this drone withou an issue quite long time. I am using GPS that has QMC 5883 chip on it.

I am not a expert on analyzing logs but I couldn’t figure out the exact reason of the crash.

What actually happened at field was dron was doing RTL and it started to descend as it suppose to be and then it fell like a rock. the rescue parachute was not even triggered.

I was late to see this post to upgrade to new version as I see now. So if this crash is related to same subject then I know what to do now. If not it will be nice to know the exact reason.

Thanks for helping…

Log is here

X1Aero · October 23, 2019, 7:29pm

Just had a quick look and it seems you’re batteries were at 3.1/cell when things went pear shaped.

xfacta · October 23, 2019, 8:11pm

I agree with X1Aero, the battery voltage is suspect - I’d say throw away that pack or only use it for bench testing

And the Vcc power supply for the flight controller is not very good, and goes bad at the end

I’d say you need a better regulator for the flight controller, it should output 5.1 to 5.3 volts consistently and not be affected by the battery voltage like your existing setup is.
4.7 volts is getting into falling-brick territory.

Maybe examine your wiring, wire sizes and connectors.