I have been developing mainly on companion PC, so I have been flying my octa with a one and a half year old release: APM:Copter V3.4-dev (78442b7e). I know, it’'s not even a stable, but we have flown probably 3 hundred hours without any single glitch. All of the sudden the system just committed suicide and I would like to understand why. Furthermore if this can be of any help to developers, even better.
My setup: Pixhawk 1, GPS M8M with magnetometer. Both the Compass were selected as in use.
The facts: last day I flew a total of 4 missions, both in Loiter and Auto mode. The first one went smooth. During the end of the second one I had some “high compass variance”, even though the system was flying smoothly. Before the third flight we calibrated the compass again, took off and still high compass variance, but the UAV was fine, the heading was correct no visible issues. I removed a component that I thought it might interfere and tested again.
Took off in Alt hold stabilize and the system immediately looked weird and not responsive, like the attitude was wrong. I set in AltHold to try and recover it but at that point the system started to roll and pitch and then it flipped and crashed. It was not a mechanical issue, the EKF variances were all red and the attitude completely wrong.
Now, I understand that everything point to the compass, but the question is how the EKF can be driven so crazy by the compass, even though the sensor is tagged with an high variance, in a manual mode. And why the system did not selected the backup attitude estimation when the primary went bad.
It does look like the attitude is going bad. In particular the two AHRSs don’t agree on the attitude. I’m not sure why so I will ask Paul Riseborough.
Just to point out a few things that you’re probably already aware of:
it’s best to stick with the official releases. From your comments above it sounds like you already know this but official releases have gone through lots and lots of beta testing while “Copter-3.4-dev” is just some version between Copter-3.3 and 3.4 which could certainly have issues. Using a non-standard version makes support difficult because we can’t know easily what’s included or not. This particular firmware is from Mar 15th 2016 so well before beta testing had started so it’s certain many issues would have been found and fixed after this.
Compass arming checks have been turned off. It’s best to leave these on and if the dreaded “compass inconsistent” message is appearing and can’t be resolved by moving power wires etc away from the flight controller, try disabling the internal compass. Disabling the internal compass disables the ability to check the external compasses orientation is correct… which is not good but if the internal compass is way off, it can’t be used to perform the check in any case.
the COMPASS_ORIENT parameter is set to “4” (facing backwards). You’re probably aware but just in case.
Thanks for using ArduPilot and it’s great that it’s mostly been working well for you. I’ll ask Paul if he can investigate but it’s a non-standard version so I’m afraid I can’t make promises…
I don’t believe it was compass induced. It looks like the IMU1 and IMU2 inertial nav solution calculated by the dual EKF’s went bad. The IMU2 inertial nav solution was noticeably worse. Possible causes for this are high frequency vibration at multiples of the sampling frequency causing aliasing and interruptions to the EKF update causing IMU data to be missed.
From the log file you can obviously see that your magnetometer data is abnormal, but you made a mistake. In the case of a magnetometer error, switch to the althold mode, the posture difference becomes larger, resulting in the EKF operation out of range. The final blower.
Thanks guys for your help. I am developing on companion pc for precise landing and basically that version was the first one that worked fine so we have been sticking to that since then. Actually recently we tried the 3.4.6 but it is not as responsive to velocity reference messages as this one. But that is another story.
The compass is oriented 180 deg due to cable shortage, the motors are quanum 330 and 17 inch accurately balanced quanum props. The tarot frame though vibrates, and even mounted on a damping bed I still have 30,35% vibes on z. It’s been like that for almost two years.
The compass check at takeoff is my fault, I did not notice it.
Now I am out for a week, but I coul post even the previous twlemetries, where was flying just fine. Then it started crying for compass variance and the bad thing is that the crash happens after a calibration.
Anyway, even though the release is not official, it still might be of some interest to developers.
About the emergency mode, I always tought alt hold or stabilize were safer than any other else. Which mode is best to switch in such a situation?
As long as the abnormal situation, the navigation, GPS, magnetometer errors, should try not to rely on a variety of sensors continue to fly, you can choose manual mode, do not use fixed-point mode. So that only the largest safety factor landed back.
