Unstable EKF variances - Velocity and Compass

ArduCopter Copter 4.4
,
1

Hey,
We’re dealing with a long-time issue of EKF variances that sometimes cause failsafe situations.
Almost every flight we see GPS Glitch or Compass error in the messages tab in Mission Planner.
Looking at the EKF innovations window we see high innovations in compass and velocity, and sometimes even in position.
We tried using better sensors, using better placing of sensors (as much as we can with our platform limitations).

Today we tried flying without a compass connected and also disabled GSF operation, and innovations were still high. Received GPS Glitch or Compass error as well (see log).

Some of the error messages we see:
GPS GLITCH OR COMPASS ERROR
IMU0 EMERGENCY YAW RESET

Is it possible to determine the sensor that cause these errors?
Our IMU is off-board, an ADIS16507 connected to the external SPI bus of our mRoPixracerPro.
We’re suspecting either bad parameter settings, or hardware. Problem is we can figure which one is at fault.
How can I determine if we have bad parameters, bad connection to one of the sensors or bad sensor (less likely, since we replaced most of them and the problem repeats)?

Logs:
compass-not-connected
compass-connected
bonus

Thanks!

2

You should start by updating to ArduCopter 4.4.1. Acording to your post you are using ArduCopter 4.0.x.

3

Wrong tagging in the post, we’re using 4.4.0.

4

Anyone?
Seems odd no one has anything to add, both here and at the discord server.

5

So a magfit calibration using the Lua script

6

Thanks,
I am genuinely asking - is there a reason to do so even when I get these messages without any compass connected?

My suspicion is IMU communication issues.

7

Arducopter needs a compass

Can you test on other hardware?

8

Yes - before upgrading to PixracerPro we tested with other flight controllers, but mostly using GSF

9

I just tried looking at the log Compass-connected, but there’s not a lot of compass data. At least there wasn’t the data points I was expecting. I might be missing something but I’d try to verify the compass is actually connected properly. Re-do the basic compass calibration outside with a good GPS lock. Then fly it around a bit to do a magfit calibration.

10

It was GPS GLITCH message, but because most cases of GPS glitch errors are caused by incorrect compass settings or compass errors it was rewritten in this form. In your case it has nothing to do with a compass.

12

@Eosbandi
Thanks, do you recon it has nothing to do with compass due to bad compass connection, or it was actually a GPS glitch?
I out area GPS signal is not optimal but should be OK to work with.

@Allister
Thanks, maybe the lack of compass data in the log is because the compass is a Matek AP_Periph 3100 connected via CAN?
I also just noticed that LOG_BITMASK’s compass field was not checked.

13

Either gps glitch, or a bad imu data, which indicated movement.

14

Thanks.
I’m more worried about about bad IMU data since GPS locations are pretty much OK.
I’ve had issue with the same flight controller and other IMUs (though they were lower grade) and lower grade flight controllers with the abovementioned lower grade IMUs.

I’ll do some flights with raw imu logging (was not enabled until now) and we’ll if this helps to understand the issue.

Here a log file with raw_imu and compass logging enabled.
Be aware that there weren’t too much of significant variances since we can only fly relatively calm in that area.

15

Do you refer to this?

16

I ran magfit on that flight and here’s the values that came from it.

COMPASS_OFS_X 60
COMPASS_OFS_Y -26
COMPASS_OFS_Z 49
COMPASS_DIA_X 1.068
COMPASS_DIA_Y 1.200
COMPASS_DIA_Z 1.050
COMPASS_ODI_X -0.003
COMPASS_ODI_Y -0.004
COMPASS_ODI_Z 0.108
COMPASS_MOT_X 0.621
COMPASS_MOT_Y 0.641
COMPASS_MOT_Z 0.423
COMPASS_SCALE 1.00
COMPASS_MOTCT 2
17

Yes, that is what Allister and I are talking about.

18

@Allister @amilcarlucas, thanks.
Will try these values.

19

Update -
Tried flying with the magfit values provides, still saw relatively high variances, but this time mostly in the Velocity bar.

We also tried switching to EKF2 following some advice from a friend who used EKF2 to solve an issue in the past (I’m not saying that is best practice). Still the variations were high.

Logs:
magfit-script-values
EKF2

EDIT
Two additional files, where the pilot had to switch from Stabilize to Acro when the drone’s horizon drifted.
With EKF2 - first Acro switch was preventive, second was pilot’s decision to avoid a crash.
With EKF3

Currently my next step is to shield the external IMU cable. Updates to come.
UPDATE 1.0 shielding the IMU SPI cable did not help.
UPDATE 2.0 Flying with the original IMU sensors, with the original onboard compass, and we still get errors. log

20

Please check your vibrations: a vibrating (also a “floppy”) frame would mess up with the IMUs accelerometers and gyros which would lead to high innovations. If you have multiple IMUs, it’s worth checking if one or the other are worse and make the good one primary.

21

We do have a “floppy” frame - it’s an H frame (not “Ardupilot H-frame”, the cross beam goes from left to right and the motor beams are front to aft, so it is a Quad-X as far as Ardupilot FW concerns).
We have only a single IMU, usually, but as you a see in prior comments I tried flying with more IMUs and had issues.

Anyways vibrations look pretty OK:

image
image1512×453 113 KB

This is a screenshot from the log file I sent here as “With EKF3”

Maybe another thing that might affect is filtering such as H-notch?
Should I be looking at other log fields in regard to vibrations?