Finding the root cause of EKF3 lane switch due to yaw innovations

Flying one of my standard test missions today, my copter got an EKF3 lane switch.

The copter continued the mission successfully - and I repeated it again right after this mission concluded - no further problems.

I would like to see how much I can learn about the cause of this lane switch. - to see if there’s anything I can do to prevent it’s reoccurrence.

I first looked at MavExplorer EKF3/Normalised Innovations and found that there was a spike in XF4(x).SV at exactly the time of the lane switch: 14:38:14.632

Using Mission Planner I found that all three EKF’s had a spike in SV at the time of the lane switch.

From here I need some guidance.

Since all three EKF’s had the same spike in SV, does that mean that the lane switch accomplished nothing - the firmware simply tried a different lane to see if another lane was OK?

(If so - I suppose no lane switch was necessary - but probably good to do anyway.)

I’m trying to understand if any of this points to a likely hardware of software problem. Do innovations on all three EKF’s generally point to software or hardware issues?

If it’s hardware - I’ll try to fix it by repair or replacement. If it’s software, perhaps the DEVs would like to take a look at it.

I’d appreciate any advice or input on next steps I can take.

Thanks you!

ADDENDUM

Since I repeated the mission directly after the one with the lane switch - you can see the effect by comparing the two flight path maps.

The lane switch occurs as the copter approaches waypoint 20.

This is the flight path on the following flight with no lane switch.

Those paths look very good apart from that glitch.
I couldnt see what caused it, though many others may be able to.

I ran magfit over that log and your compass settings are pretty close, you could adjust them to these if you want to be a bit more exact:
COMPASS_OFS_X -91
COMPASS_OFS_Y -67
COMPASS_OFS_Z -7
COMPASS_DIA_X 1.033
COMPASS_DIA_Y 1.014
COMPASS_DIA_Z 0.897
COMPASS_ODI_X -0.029
COMPASS_ODI_Y 0.072
COMPASS_ODI_Z -0.129
COMPASS_MOT_X -1.215
COMPASS_MOT_Y -0.061
COMPASS_MOT_Z -1.027
COMPASS_SCALE 1.00

COMPASS_OFS2_X -226
COMPASS_OFS2_Y 30
COMPASS_OFS2_Z 254
COMPASS_DIA2_X 0.985
COMPASS_DIA2_Y 1.053
COMPASS_DIA2_Z 0.800
COMPASS_ODI2_X -0.013
COMPASS_ODI2_Y -0.013
COMPASS_ODI2_Z -0.016
COMPASS_MOT2_X -2.408
COMPASS_MOT2_Y 0.882
COMPASS_MOT2_Z 1.608
COMPASS_SCALE2 1.00
COMPASS_MOTCT 2

and unrelated but attitude control looks really good.

1 Like

I experience the same EKF3 lane switching today but I can’t find the root cause.
There are log and the snapshot of the log below

Hope you can help to analyse.
Thanks.

Hello -

I’ve had several lane switch occurrences in the past couple of years.

I’ve never been able to discover the root cause of any of them. I suspect some “glitch” was the cause for each occurrence.

It appears to me this is the idea behind the creation of EKF3 and the ability to lane switch. As a “glitch” may happen at any time - the ability to lane switch provides a safety net.

Where possible, I use a telemetry connection to Mission Planner as part of my ground control - and I bring up both the EKF and VIBE displays so I can monitor them for problems as the mission progresses.

My hope is this will allow me to tell the difference from a “glitch” and a “failure” resulting in a lane switch - by watching for steady rises in the displayed innovations.

To date - I’ve never observed such a steady rise in EKF3 innovations - so all of my lane switches appear to be “glitches.” But I investigate each one anyway.

If I ever observe a steady rise in either EKF or VIBE in the displays, I’d hope to abort the mission before a lane switch occurs.

I cant easily see why there’s lane switches, but I suspect it’s related to vibrations (which are within limits but borderline) and tuning. There’s some attitude oscillations and a bit more noise in the motor outputs than desirable.

Update to latest stable firmware since there is important fixes for Cube Orange.

Before improvements I wouldnt be doing such long flights or Auto missions until you’ve finished sorting out the basics, like some extra attitude control tuning.
Also it looks like you’ve got 3 GPS units which is a bit pointless since only two are used.
Keep the best two, it looks like only one has a compass, so keep that one.
Check the compasses are all present and accounted for.

Start with these, some are only a tiny shift from what you had, but could help

ARMING_CHECK,1
FENCE_ENABLE,1  // check other fence settings
GPS_GNSS_MODE,65
GPS_GNSS_MODE2,65
INS_HNTCH_FREQ,33
INS_HNTCH_BW,16
INS_HNTCH_FM_RAT,0.7
INS_HNTCH_REF,0.14
LOG_BITMASK,145406  // cut back the logging a bit
MOT_THST_EXPO,0.4  //  you had instability at low throttle, descents

Do a test flight in AltHold with plenty of yaw, circles and a figure-8 if you can. It only needs to be a short flight. Then we can run that log through magfit to really fix up the compass parameters.

PIDs look a bit odd, like D terms are too high. It would be good to do Autotune one axis at a time.

1 Like

Thanks Shawn for pointing out valuable points.

I used 2 GPSs, the third one seems to be the blended one

I just use one external RM3100 compass, the internal one (inside the Cube) is ignored. My system is not using Compass + GPS combo like the regular one

It should be 31 on the no-payload flight
In this flight, it carried a payload

A normal Tmotor P80III + Alpha ESC, do you think we really need to change it from 0.65?

This is not my only or rare long flight but I will definitely do some fine tunings later on.

By the way, it is really helpful if you have some time to point out the reason of EKF lane switch.

Ah yes - you must be right!
It’s my mistake.

Update:

In the second time Lane switching from 0 to 1, there was a small oscillation at Pitch axis and X axis had a spike of vibe

Yes, apart from some history of people using 0.4 successfully on other similar builds I’ve observed, you had some tell-tale signs in the log. Like instability as soon as you start to descend and the rule-of-thumb for setting and testing MOT_THST_EXPO is

  • set too high you can see instability at low throttle
  • set too low you can see instability at high throttle

Also if you really want to check that expo, and it would be a valuable exercise for all of us - try this thrust expo script:
https://discuss.ardupilot.org/t/automatic-mot-thst-expo-estimation-lua-script

I was just being a bit too picky with suggesting INS_HNTCH_FREQ,33 wasnt I? :slight_smile:

1 Like

Just seen that yesterday and following the progress. Tridge did amazing work.

Definitely not, this drone is crazy and 33Hz works sometimes.

By the way, I am still diving more in the cause of lane switch. It must be a “decision function” where it results the change in EKF lane used.