We experienced a crash recently on a copter using ArduCopter 3.5.4 on a Pixhawk. We had a GPS glitch (which last for about 14 seconds) while flying in stabilize. Then, a couple of seconds after the primary EKF switched multiple times (see Figure 1).
This caused major oscillations on the copter and we finally crashed. We found that there’s a significant delta between the estimated attitude of both EKFs. So, switching from one to the other causes significant discontinuities in the actual attitude estimation. Figure 2 shows the estimated pitch for both EKFs and we clearly see that there’s a drift between them. Indeed, looking at our previous dataflash logs, we observed that most of the time.
GPS Glitch can be logged for any number of reasons. It will cause the EKF processes to switch just about every time. I tested this quite a bit with a helicopter set up for full manual control in Acro for bailout, and flying under a 250kV interstate transmission line to get a GPS glitch and see what the EKF “lane switching” does.
My solution was to disable the extra EKF processes. Getting rid of the extra EKF processes, I found, will still cause the attitude solution to go bonkers when you get the “GPS glitch”. But the helicopter remained controllable when it can’t switch to a different attitude solution via another EKF process.
Until the over-reliance on GPS is designed out of the system, and GPS is only used for horizontal navigation (GPS is not even reliable on altitude), there is not an easy “fix”.
The reason your attitude solutions are significantly different is because the accel’s don’t agree. If the IMU’s don’t agree, the attitude solutions will be far enough off to cause quite radical upset when it switches
Thank you very much @ChrisOlson for your analysis.
Regarding the difference between the two EKF, my first idea was indeed the difference between the accelerometers. However, for me the measurements are coherent. The second IMU is noisier than the first IMU, but the agreement between them is still good.
What tells you in this log that the two accelerometers do not agree?
The vibes looked good to me. But unless those IMU’s agree exactly, I’ve found quite a bit of difference in the attitude solutions if the EKF switches. It’s easier to see on a heli because the servos go wild when it switches. And in my experience a noisy IMU can cause it.
OTOH, when the accels agree right on, the EKF can switch and you can’t even tell it happened. It is possible you have a hardware problem there. I’ve had some cheaper Pixhawks that show the type of IMU noise/aliasing that yours shows in helicopters, and those have been really interesting to fly when the EKF suddenly decides to swap lanes.
GPS glitch can be logged for various reasons, but it usually means the GPS position has become unreliable for some reason. The system is designed to cope with that, but it isn’t necessarily designed to cope with hardware that is not working properly. When you have dual IMU’s they really should both show the same thing. It might be worth it to try a different controller and see if you can get rid of that IMU noise.
Looked at your data @Phil007,
you certainly get a big drift in pitch and roll during the flight. looks as the gyros do have difference from the beginning, you should check for the reason.
I’ve checked my data, non of my sets had a difference of more then 1.5 degrees,
yours go from 5 to 12 degrees. check the AP …
