Compass issues - the dreaded toilet bowl!

Hi all I have a bit of a head scratcher (at least to me).

We operate a large octocopter that we have been running with Cube black and the spektreworks carrier board with a Here GPS for sometime and haven’t had any issues with the compass up to this point. we recently updated to Copter 3.6.10 and now we have a weird intermittent compass issue that causes the dreaded toilet bowl.
We calibrated the Here compass (both 1 and 2) both passed no issues and and flew number of successful flights we then had a toilet bowl which lead to a crash.
We fixed the damage and it flew fine a few time then we had the problem resurface. However it is now more persistent.
To start with I turned off the internal compass (number 2) and still the issue persisted. I then turned off compass 1 and while mission planner is showing the error “bad compass health” the aircraft flys fine once a 3d fix is achieved.
So I started a process of elimination initially suspecting that it was the Here unit that was faulty (it was one of the first i ever bought and it has had a hard life) so I put a Here2 on and ran a compass cal (which was successful). Here is the kicker I got the same behavior including the deactivation of compass 1.
I have uploaded logs while running the Here 1 and the Here 2 to a google drive folder and the link can be found at the bottom of the post.
Any input would be great as this one has me stumped!
https://drive.google.com/open?id=1HDShFcJ9DIizL3PvcVVM5ZdC3MWl89gv

I’m no expert, and this might be completely wrong, but generally I start by checking out vibrations before looking at anything compass or IMU related. The UAV has IMHO quite high vibe levels, especially in x and y (try to get them below 12 m/s/s). This can affect IMU accel interpretations which in turn can affect the interpretation of the compass. So one thing to do might be to try to reduce the vibes to give the EKF2 cleaner signals.

Thanks for the response. I knew we had a fair amount of vibration but I wasn’t aware that it could effect the compass. I am going to do some testing on 3.6.11 as advised by the dev team as there are some issues with i2c in 3.6.10. if that doesn’t cure the problem then i will vibe isolate and try that.

Checkout: http://ardupilot.org/copter/docs/common-measuring-vibration.html

Vibration doesn’t effect the compass. But because many sensors are fused and it effects the IMU it can have wide ranging effects. I’m not necessarily suggesting vibes are your problem as there is not large amounts of accel clipping just a comment. In the log I looked at (out of the many in your link) you had compass 1 inactive and compass 2 active so it’s using one of the internal compass’s. That can be problematic…

Hi Dave,
Thanks for that insight. The log you are referring to is where we turned the primary off and the toilet bowling ceased when we did this. We are getting the issues when compass one is enabled which is very strange as it is the external… that why i posted this as the normal culprits do not seem to fit.

While the first Mag looks ok


The second and third Mag are VERY noisy by comparison

Do you have high current wires near your FC?

The other thing I found curious was the 1300PWM bottom limit you seem to have on motor outputs.


Any particular reason for this?

As mentioned previously your Vibes could do with attention.
It’s not so much the absolute magnitude but the fact that clipping is occurring.
That needs attention.

As I have said before the first mag (the one that is less noisy) is the one causing the issues. When i disable that mag the aircraft does not toilet bowl.
Phillip Rowse had a look at the vibration and it is within acceptable levels and is not clipping. I will be running an FFT log to confirm that there are no harmonics.
The PWM I dont know for sure whats going on other than the spin mot min that has been set for when the aircraft is armed. As it effects all channels i’m guessing this is the case but I will have a check just to make sure.

I can graph compass noise, thats the first thing that I did as a noisy compass 1 would be a smoking gun. But the way the aircraft behaves when running on the far noisier internal compass systems is ironically better ergo there is something deeper going on here.

So i have completed an FFT log for the vibration and a compass mot.
Turns out my issue was the amp variation effecting the external compass in waves. The compass mot and new shielding has resolved the issue but the weird part that was still bugging me was why it had chosen now to present. The answer to that one was looking at historical logs and comparing the average current draw. Turns out the draw has gone up as the motors have aged. What was borderline initially was pushed over the edge as the current draw went up.
I am still looking at the vibrations to make sure that they are within happy limits.
Please see my FFT result below can anyone help me interpret them?