Plane 4.4.0 on Pixhawk 4 BMI055 data corruption + LaneSwitch leads to crash

[YupsUAV]

Hello!

I’ve found the cause for a very serious crash i had for the second time now.

I’am using a Holybro Pixhawk 4 (dual IMU) with M8N GPS on a Crosswind Mini Plane, single Motor.

The crash happened with 4.4.0 two days ago and on 4.2.0 in the Summer 2022.

Root cause is a defective BMI055 or even a driver bug when reading from the BMI055.

The plane was on a mission without any notable problems running on first IMU (0). During final dynamic descent, Arduplane did a LaneSwitch to second IMU (1 = BMI055) which has its Gyro Axis mixed up since flightminute 5. Orientation was ~YAW180°, this lead to a total loss of control. I wasn’t able to switch to manual to get it back into my control but hat some luck to avoid a nose landing, so it “just” broke the fuselage once…

Took a while to finally find these problems. @Rolf helped a lot in a local community.

Ground Track of the landing approach, Plane came from south with headwind. POS and GPS difference!

Attitude Control: You can see the moment of the LaneSwitch shortly before the cursor. Roll and Pitch estimation suddenly change dramatically.

Gyro Comparision. Second Gyro Axis are mixed up:
X->Y
Y->Z
Z->X

Moment of the second Gyro AXIS-SWITCH at 4.53.5

Another example of a longer flight:

There where 3 Gyro switches in this flight. First to the above variant “A”, later to another mix up “B” and back to the correct squence.

Honestly there are 2 big problems:

• A Hardware defect on the IMU or a driver bug.
• A very bad EKF3 behaviour which switched to the bad IMU although it was already marked as unhealthy.

I hope some of the devs will find a solution for this. I can provide several original logs per DM.

Here is the recent crash log with random GPS:

00000022 -anon.log

Thanks and best regards,
Yannick

[Rolf]

The sudden output of gyroscope values on the wrong coordinate axis is unbelievable. I had considered whether this was just an error in the log file recording, but the other values such as roll are also correspondingly incorrect. This is beyond my scope.

Two gyros, initially, the measured rotation rates of both gyros matched well. Therefore, the red plotted IMU[0].GyrX is covered by green plotted IMU[1].GyrX until time A. From then on, IMU[1].GyrX deviates completely from IMU[0], easily recognisable by the red area:

Therefore e.g. wrong roll angel by IMU[1] from time A on:

Strange: IMU[1] actually measures correctly, but the GyrX values are output as GyrY. GyrY becomes GyrZ and GyrZ becomes GyrX from time A on.

IMU[1] gyroscope is continuously flagged as healthy (IMU[1].GH = 1) and the crash occurs after Primary Core Index XKF4[1].PI for another reason switches from 0 to 1 at around 14:13:35 thus to the gyro with incorrectly labelled axes.

Rolf