EKF3 yaw estimate deviates on different instances on second flight only

Hello,
I have several identical drones (cube orange copter) that sometimes have this issue. The first flight goes without problem, but on the second flight 20 min later the yaw estimate on different EKF instances starts to deviate. This leads to toiletbowling and in some cases a switch to land mode. No reboot was done in between flights.

If this behavior happens on the first flight, I would just assume the sensors need calibrating. But I am wondering if there is something more at play, since this happened directly at the start of the second flight.

Since the EKF3 Yaw inconsistent prearm check was not triggered, is there a way to detect this behavior before arming? Are there perhaps some parameters that are incorrect?

D240.param (18.0 KB)
logs: Unique Download Link | WeTransfer

An update on this issue.
The drones are now running copter 4.4.4 and the issue is still common.
A compass calibration was done before the flight.
EK3[1] is primary, EK3[2] is using a hard mounted IMU and has more vibrations.


uwsoftware.be/2024-10-31%2011-51-35.bin

Some extra observations:
-XY0[1].YC (GSF output) is close to the actual YAW observed during the flight.
-AHR2.Yaw seems to follow XY0[1].YC with a constant offset and could be used for flight.
-The raw compass readings seem to be correct and are indicating the heading observed the during flight.
-IMU 0 and 1 are healthy with vibe < 10, low position innovations also seem to indicate a healthy IMU.
-ATT.Yaw is constant because the copter is trying to maintain this heading, but in reality it is chasing the wrong value.
-xkf1[1].Yaw is the same as ATT.Yaw which leads me to believe it is using the estimated YAW value as the true value used for navigation.
-The yaw estimation improves when the GSF uncertainty improves during the lateral movements when toilet bowling. But once the toilet bowling stops, the error increases again at an almost fixed rate.
-EK3[0] and EK3[2] are similar, but initially deviate in the exact opposite direction.

These are the questions I still have.
-why is the yaw estimation incorrect? This leads to high innovations and eventual rejection of good sensor data.
-could the almost linear gradual increase of the error be due to an integration error?
-why is there such a large difference in between the different EKF cores and AHR2?
-why does this only happen during the third flight?

Since this issue was reproduced on copter 4.4.4, @amilcarlucas can this topic be moved to copter 4.4? I don’t seem to be able to do it myself.
Thank you.

Just to summarize. The drone is toilet bowling and we don’t seem to understand why.
Drone is calibrated correctly, we even added z-throttle magnetometer compensation.

How sure are you that all has been done correctly and in the correct order?

Do you have checklists? Do you have a method?

Do you know that a recommend method and checklist exists? Automated by a software used by hundreds of users.

We have been using ardupilot since 3.5.8 and a lot of flights and experience (drone shows) under our belt. Ardupilot is really high quality software!
Feel free to give a link to the software. We are always looking to improve our methods!

Normally we can pinpoint issues to something (hardware / configuration / calibration / human error). But here we cannot find how to proceed.

We see toilet bowling, but the yaw values seem to be correct. Only EKF3 thinks it is not?

It is available in english, chinese and portuguese.

1 Like

For enterprise users like you, with hundreds of vehicles, not using the software should be considered a crime. It’s very easy to miss something when not using it.

And see section 9 of the user manual.

We just went through it and my impression is that this is good for a first user or new hardware configuration. Nice work! Though we already have a system in place to store the params (between different ardupilot versions and different hardware) and make sure all drones have the correct parameters at all time.

The tool didn’t mark anything, sadly.

Going deeper the rabbit hole in the log (http://uwsoftware.be/2024-10-31%2011-51-35.bin):

  1. AHRS2: looks to give the yaw we observed during flight

This is the DCM, iiuc, (magnetometor only)

  1. EKY0: looks to give the yaw we observed during flight

This is the GSF, iiuc, (everything without magnetometer)

So all sensors agree on the direction the compass is going. Except XKF1 (EKF3) thinks on all cores that we are rotating the other way around ?!?

Interestingly after using “Replay” on a log with LOG_REPLAY=2 and LOG_DISARMED=1 the XKF1.yaw is correct when I use EK3_MAG_CAL=2 .
Not sure which conclusions that gives …

(The first flight of this log Dropbox )

I’ve been playing with the parameters and replay. But very confused by EK3_MAG_M_NSE and EK3_YAW_M_NSE. What is the difference?

The values seems better with:

EK3_YAW_M_NSE: 0.1 (instead of default 0.5)

Am I correct that this is only used when not fusing IMU / GPS and compass together. E.g. at yaw reset? This gives a lot extra confidence to compass. Could it be because GPS has no heading yet, since it hasn’t moved in a direction yet? This helps predominantly at start of a flight.

EK3_MAG_M_NSE: 0.12 (instead of default 0.05)

What does this do? This seem to help towards the middle/end of the flight.

  • Before: green, orange, red
  • After: blue, pink, purple
  • Compass: black

NOTE: during the log we have not moved in X,Y. Only hovered. When moving we should trust IMU/GPS more. So these are not ideal values.