Hoping someone with strong EFK knowledge can help out here @rmackay9@Paul_Riseborough . Yes, I am still on 4.0.3 using EKF2, but I believe this issue would still be relevant on 4.3.6 using EKF3.
This was a very scary moment where the aircraft just suddenly went out of control and I obviously can’t have this happen again. It was a miracle I was able to save the aircraft and not hit a tree and fall in a river.
Pretext: I was flying very low, maybe about 30-60cm off the road following a car, and just as I started crossing the bridge, the aircraft went crazy (with $125k USD worth of camera gear onboard).
The event starts at 15:11:08 UTC. I am not the strongest at analyzing logs, but it appears as though a mag-realignment went bad and set the aircrafts yaw 180 degrees incorrectly which made the aircraft go crazy? My main question: WHY did it perform a mag-realignment (especially when there was mag interference from the bridge)?
One thing I noticed about this particular flight was I never went high enough at the start of the flight for it to do an in-flight yaw calibration (until it does it after the bridge). Would this have done anything to prevent it? I am assuming the bridge/rebar has something to do with this but I believe there is more to it. I have done similar flights probably a hundred times and never had an issue.
This whole event takes place in about 5 sec which started in loiter. Once I saw the craft pitch max loit angle forward I tried quickly switching into althold/stabilize but it still didn’t feel like I had any control for about another 3 sec which I don’t understand. After those few seconds I seemed to get control back with about 0.5sec to spare (otherwise it would have been in a tree and falling in the river )
Anyways, I am hoping someone can help me better understand the sequence of events that caused this to happen and if there is any way to prevent it again.
Not answering the mag realignment issue, but there are a couple of serious issues and many improvements for Cube Orange in latest stable firmware.
I would change to using EKF3 too.
It seems there’s power lines in the area and running along beside the road and crossing the road, the copter didnt seem to have issues with that.
It seems the compass2 yaw was already a bit off and maybe steel within the bridge concrete messed it up??
I would get out in a field, away from the road and power lines, and do a flight with circles and plenty of yaw. Then use magfit (offline) to fine tune the compass calibrations, and this will also sort out the compass/motor params.
I got these from magfit on your log there, but they might not be appropriate since there was an issue with that flight.
Thanks Shawn, I am in the process of updating our aircraft, but I don’t have enough testing on the newest firmware to put them in operation yet. This has been our tried and tested craft and have been flawless for the last two years until now. The mag cal maybe could be a bit better, but during normal flight away from interference sources it flies perfectly straight.
I think I may have discovered a rare but potential dangerous issue the more digging I do. It’s doesn’t appear to be resolved in the latest firmware / EKF3. Hopefully one of the developers may be able to shed some insight on how this happened and how to prevent it (or fix it).
If there is an unfixed issue at play here I would still urge you to update to latest stable (and change to EKF3) as there are definitely (potentially) serious Cube Orange issues fixed in recent releases - so at minimum you wouldnt be making the your mag issue any worse. The new versions and especially EKF3 has up lots of testing and flight time.
I’ll check over you log again to see if I can find out more.
EDIT
I can see where all the bad stuff happens, magnetic field changes and innovations and so on, and the yaw innovation flat-lines, so it seems like yaw (and other attitude control) is handed off to another EKF instance but the logging and number of IMUs was less than ideal in this firmware version.
For example you are only using 2 of 3 IMUs, and only logging all data for the 1st. So I think it’s going to be difficult to tell more unless someone else knows more about this specific type of issue.
What I would do from here onwards, which may seem a bit drastic, but I would be a lot more confident
Flight test, tune (should be easy as it was flying well before)
Magfit
There’s some things I would probably adjust right now too, like
INS_HNTCH_REF,0.39 should be 0.245
and I can help with the parameters to put back in place if you go down that route.
)