Crash analysis please? Crash after errors: velocity variance, compass variance, pos horiz variance & EKF

Had a strange crash today with a very well flown plane, she’s had at least 200 hours flight time since 2015 with no problems and now suddenly it crashes.

Was flying around in Training mode teaching a student and started getting errors: velocity variance, compass variance and pos horiz variance. Also got “EKF IMU0 yaw aligned to GPS velocity” message.

It pitched down and I switched to FBW-A mode but no response, it kept pitching down so I switched to Auto while holding full up on the elevator but she continued to dive and T-boned a fence.

Right at the end of the BIN log the ATT.DesPitch goes to +15° but pitch is -20° with RC.OUT at 1478 (which is elevator up) but it didn’t pull out of the dive. Post crash, the elevator servo is working fine and linkages 100%.

Any idea as to the cause would be greatly appreciated.

BIN and TLOG below:
TLOG: Dropbox - 2021-12-09 08-28-43.tlog - Simplify your life
BIN: Dropbox - 00000110.BIN - Simplify your life

From what i can tell, the flight controller suddenly somehow thought -20 was level, so when you were in FBWA trying to pull up it thought it was already level and your were already trying to pitch up, when watching the tlog in mission planner you can see the plane clearly thought it was level all the way into the ground.

Its possible it’s a electronics/software problem thats reset it but the only other time I have ever had something similar happen was when my flight controller came loose inside the plane and shifted 20 degrees nose down, this made the plane fly 20 degrees nose up. are you sure the flight controller was secure?

the way I got it back down was to use stabilized mode as it has no pitch or roll limits, so I could hold the nose where I wanted it. manual mode and acro would also work since they dont auto level.

Thanks, yes the FC is/was secure, it’s held by strong velcro on a balsa base (to which it’s still attached in the wreckage). Unfortunately there wasn’t much time to switch to Manual and as FBW-A has always been 100% reliable in more than 6 years of flying so I didn’t even think to switch over to Manual.

With all those errors reported I think something went wrong with the FC’s attitude estimation. What, I don’t know, hoping someone here can tell me.

its really strange, from what I can tell all the sensors were working fine. you had gps, voltage levels were fine.

what flight controller are you using? I notice it only has a single IMU? is it a f4 racing controller?

It’s an AUAV-X2 from a few years ago (Introducing the AUAV X2.1 Flight Controller - Blogs - diydrones) and has been faultless up 'til now.

I think the IMU has glitched and its lost its orientation, on a pixhawk it should just switch to the second imu and keep going.
you can see it loose attitude seconds before the EKF errors flash up. it shows the hud at 20 degrees nose up then it suddenly resets to level when you select FBW-A.

Something is really wrong with your controller, your missing an IMU, there should be IMU0 and IMU1, when the first failed it should have switched to the second one, but for some reason your controller is only showing one active. I thought you were just running a f4 racing controller because there is only one in the log.

it should have both
Invensense MPU9250 9DOF
Invensense ICM-20608 6DOF

what version of arduplane are you using?

Arduplane v4.1.4.

Strange that its not showing the second IMU, I never noticed that before and as the plane has been flying so well I never bothered to look.

If both are glitching I would suspect the 3.3v rail on the flight controller, I have only seen imu fail in flight once on an old apm 2.5 when the 3.3v regulator failed.

go back and check the oldest logs you have and see if both are showing up. It would be interesting to see if the IMU failure was limited to that flight or how long it has been an issue. Its possible it never worked from the factory it would just fly like any other f4 controller with a single IMU, the plane wouldn’t have ever shown symptoms until the remaining one failed.

check the board for corrosion too, its usually a common culprit.

So I’m not sure how to check for the 2nd IMU but looking at a logfile from 2015 it seems only one IMU is present.

IMU

yes it should say IMU and IMU2 . looks like it never worked.

@rmackay9 are there any checks on startup to see if all the IMUs are present or does it just go with what it detects on startup?

i know this comment is of little value for the present situation, but it may help others/later for me to mention that this illustrates why i will switch to a manual mode in the case something goes wrong (if i can see the plane or camera view). i am confident in my ability to fly the plane manually so, in the event of a hint of FC or flight control malfunction, i want to take the potential problem out of the control loop as much/soon as possible.
of course, a malfunctioning servo, bad power, loose control rod/cable/fitting, airframe failure would still cause trouble.
i always have my “manual mode” switch in an easy to access place on an RC xmitter and i practice using it. i dont guess there’s an option to fly manually with just a ground station?