Loss of correction after switching couple times between yaw compass source and GSF source

Good afternoon,

Today we tested the Gaussian sum filter as a source for yaw. We are very impressed with the quality of the estimation since the drone indeed performed very good when we switched to loiter mode after warming up the algorithm in altitude hold. We felt very confident with its performance that we decided to test the transitions, manual source switching with a Lua script. I daresay that the transition between the source 1 (GPS position/ compass yaw) and the source 2 (GPS position/ GSF) was pretty smooth. However, after switching a couple of times between lanes, an EKF fail-safe was triggered. The sequence was this:

  1. Take off in loiter mode using source 1 (GPS/compass).
  2. Switch to source 2 (GPS/GSF) after let it warm up.
  3. Switch to back to source 1.
  4. After a while, switch to source 2.

It seems that after the second switch to source 2, the correction from GSF is not applied to the yaw anymore since the value does not change as it can be seen in the plot below:

We saw this loss of correction in two separate flights, the logs can be found below:


The switching between lanes is done with a Lua script that does something similar to what @rmackay9 is doing in ahrs-source.lua:

PS: We flew with the version 4.1.0-beta3 because we forgot to upgrade that drone, I did not see anything related the GSF in the release notes of the 4.1.0-beta4 so if there is an issue I guess it may be still there.

2 Likes

@sbaccam,

Another first! I’ve added this to the “to be investigated” list and we should be able to figure out the issue in the coming days.

As you thought, I don’t think there are any difference in the GSF or source switching logic between -beta3 and -beta4.

What do you think about the possibility of removing the compass and just use GSF for yaw in poshold, guided modes ?

Good morning @Notorious7,

Thank you for your comment! The reason why we specifically have compass in this drone is because GSF needs to warm up. You need to move it around for a few seconds in altitude hold or stabilize until it actually starts working. The reason is, as I think @priseborough explained in a presentation, that it is based on velocities. When the vehicle is idle in the ground, it has not movement for calculating the yaw.The lane switching and the Lua scripting are attractive features for not having to take off and move around in altitude hold mode. With an automatic transition from a script, you can take off in loiter with compass and eventually switch to a GSF source.

In summary, what we are trying to do is to deal with that warming up of the GSF without having the necessity to move from loiter mode. In addition, for this specific system, it will be great to have the compass as a fallback which was the inspiration for testing switching back and forth between sources.

What do you think about the idea of using compass till there is some issues with compass like toilet bowling and then GSF feature automatically takes over for the yaw measurement ?

Basically I work with smaller quadcopters which have some tendency to do toilet bowling after multiple good flights, so just wanna find out how much can I rely on GSF for yaw in that cases, I am not sure if the change of magnetic fields is a good indicator about the chances of toilet bowl cause in some cases even with less than 10-15% change of total mag field can sometimes result in toilet bowling

1 Like

@Notorious7,

Yes, GSF was originally intended to be used only as a backup heading source to catch situations like you describe. Since then we’ve made it a bit more flexible so it can be used at any time but we haven’t tested this much.

Here is a video of a test I did in which GSF is used to correct the heading after takeoff.

1 Like

I have also tried something similar to what @sbaccam done in his case, i’m running arducopter-4.1.0-beta5 on custom build FC similar to omnibusf4pro, i went for changing EK3_SRC1_YAW parameter during the runtime using rc switch, flying in poshold mode only, i reproduced the toilet bowl effect on my quadcopter by slightly changing the compass orientation after calibration, switching EK3_SRC1_YAW from 1 to 8 seems to worked quite well so far, even switching EK3_SRC1_YAW from 1 to 0 also falling back to GPS/GSF for yaw, toilet bowling behaviour stopped pretty quickly most of the time, i am not sure about when EK3_SRC2_YAW param comes into picture, seems like it’s for when there is option for EKF lane swtiching. But i thought there might be some provision already to fallback to GPS/GSF from compass in case when there will be toilet bowling, but it does not seems like that, we have to change EK3_SRC1_YAW to 8 using rc switch to stop toilet bowling.

But still in one of the flight, after switching EK3_SRC1_YAW from 1 to 8, there was toilet bowling behaviour for quite some period even after there was a message that EKF3 IMU0 yaw alinged using GPS, i am not sure for what reason that happened even though it was not using compass, hope @priseborough can also look into it, I am attaching that log below:
https://drive.google.com/file/d/19N1rk82MjjmHnLr-zMivyGB9y4YNMhCX/view?usp=sharing

Also, as @sbaccam mentioned, there is issue with switching back and forth between compass and GSF for EK3_SRC1_YAW, i confirmed that by changing EK3_SRC1_YAW from 8 to 1, that also happened in above flight, changing the source for yaw from GPS/GSF back to Compass during runtime causes EKF variance and still there was a message EKF3 IMU0 yaw alinged using GPS

@Notorious7,

Thanks for the report and log. As shown in the video the EKF should switch from compass to GSF if the yaw is very wrong. My guess is that the toilet bowling is not severe enough to trigger the GSF but I’ll ask @priseborough and/or @tridge to provide an opinion, thanks!

1 Like

The EKF currently only resets yaw to the GSF estimate and stops using compass if the yaw error is enough to make the EKF reject the GPs data. Under normal circumstances with bad mag offsets the EKF will quickly learn the new bias values when it toilet bowls. The problem with rotating the mag is that the EKF cannot find offsets that will solve the problem for the flight duration.

3 Likes

@rmackay9 in the log you can see we did get some toilet bowling even with the yaw source set to GSF although it did get corrected in few seconds

Do you mean that even when we can see this message EKF3 IMU0 yaw alinged using GPS, EKF still need accurate mag offsets/bias values to tackle toilet bowling ? Cause me and @Deficiant_Codger didn’t get it what makes quadcopter to toilet bowl for more than 15 seconds after getting above message

Changing the yaw source parameter from 1 to 8 without re-booting has not been tested. If you want to use GPs for yaw EK3_SRC1_YAW should be set to 8 before boot, the vehicle should be flown initially in a non GPS mode loike ALT_HOLD until the GSF is able to determine the yaw. The EKF will then set it’s yaw and start using GPS, after which GPS modes like LOITER can be used.

I think if there will be some provision where user can switch the yaw source from compass to GSF during flight then it can come useful for many applications, like in our case we are planning to fly hundreds of nano quadcopters at same time in guided mode, problem of nano drones is that there’s not much space for isolating compass, compass motor calibration does helps but it still does not exterminate the chances of toilet bowling, there goes lot of effort in making sure to avoid the case of toilet bowling by running multiple test flights for each drone, and for hundreds of drones it will take a lot more effort and time

@rmackay9 @priseborough With 2 IMUs, is changing the EKF lane manually is a safer option than changing the current yaw source by setting EK3_SRC1_YAW during flight? Or if I have only 1 IMU can there be better option to switch yaw source from compass to GSF/GPS safely ? I don’t know if there’s a provision to run 2 EKF instances/cores with 1 IMU and then some safer alternative to switch between EKF instances/cores where there is different yaw source

You may want to check the Lua scripting and code something similar to this example:

That is actually what we did. The lanes use the same correction source, unless you are using affinity, which may not be the case. Basically, when you do something similar to what the script does, you are changing the correction source for all your lanes to let’s say GSF.

Thus, you can set EK3_SRC1_YAW to something like compass and EK3_SRC2_YAW to something like GSF.

1 Like

@Notorious7,

It should be possible to change the yaw source even if the vehicle has a single IMU. The sources used are used by all active EKF cores. The active source set can be changed using lua (as @sbaccam has said) or using an RC switch. In the future we will add the ability to change the source set using a mavlink message.

1 Like

Good afternoon @rmackay9,

We discovered today that what was limiting the number of times we could switch back and forth between sources when one of the sources is GSF, is the parameter EK3_GSF_RST_MAX. By default is set to 2, which explains why it was triggered after a couple source switching events. When we made it 6 it lasted six source switching events.

1 Like

@sbaccam,

Great find! So we should change this so that the reset max is only used during emergency yaw resets and not when we actively change the yaw source to GSF.

@sbaccam,

I think I have a solution here although I need to test it a bit more before we try it on a real vehicle.

EDIT: I’ve tested the PR some more in SITL and I think it is working. Txs again for the investigation, that was really helpful.

1 Like

These are great news! Thank you for solving this!