occasionally after ardurover (4.7.0 but we have been seeing this for several years across several versions) has been sitting for a while we see large position and attitude jumps. This rover has been sitting indoors with poor GPS for about 8 hours since ardurover reboot:
If I restart ardurover (to create a .BIN file, for example) it goes away. So it’s difficult to reproduce. Basically we see it after ardurover has been sitting for several hours.
Curious if others are seeing this?
I’m aware of some struggles with continuous operation where 24+ hours caused poor autopilot behavior, but I’m not aware of a specific issue with GPS. I’d assume there’s an overflow happening.
@Christopher_Milner is this specific to your Linux-run Rovers, or did you experience this with STM32-based autopilots as well?
@geofrancis any insight from your long-term projects?
I think I have seen this happen on a rover that is sitting in the clear sky with a good GPS fix. Again, only after sitting for many hours. I did see this last year when we were running on pixhawk. I too have the sense there’s an overflow somewhere in the EKF math.
I havent seen anything but i havent really been looking for it. The position jumping around when indoors is normal as its going to be getting a very poor signal.
From a GPS perspective the regular understanding is:
The longer a GPS is operational, the more precise the position information will be. (Talking outdoors).
So my money is on some sort of buffer overrun within the FC.
There are interference sources that can cause this. Try connecting the GPS receiver to a PC. Does it still jump?
If this is true than also this is the answer. Indoors the GPS receiver only saw part of the hemishere and the received signals are corrupted by multiple reflections changing over the time
Somehow I missed the indoors part. I’d like to see a log from outdoors over a similar timeframe exhibiting the same issue. Otherwise, I think we have to conclude that it’s just poor reception.
As @Yuri_Rage says a log would be good. The jumping back-and-forth between EKF and DCM is unexpected. Rover-4.7.0 should never fall back to DCM although I see this is 4.7.0-dev which could actually be quite old so perhaps it doesn’t have that fix.
One guess is that the EKF is becoming unhappy because the GPS is reporting extremely good accuracy… impossibly good. Anyway, a log might help us figure it out.
txs!
I just run a test on ArduRover V4.8.0-dev and i also can see several switches between “AHRS: DCM active” and “AHRS: EKF3 active”
But I use still my ESP32 test platform at the moment placed partly under the roof of my terrace only slightly open to a quater hemisphere
BTW, if anyone can reproduce this issue, a replay log would be really useful. Replay logs are big but in any case you can create them by setting:
I just tried to produce such a logfile:
The board was this time indoor near to a small window to the west.
Take in account, my ESP32 FC has no baro, so also the hight is taken from GPS and HDOP is poor.
It takes long to get a 3D-Fix and therefore only near to the end of the log you can see AHRS changes between DCM and EKF
I am not sure if this is also represantativ for a “real” FC
Thanks for the log. In this case the issue is the GPS’s reported position is moving around a lot presumably because it’s indoors. This is expected.
I can also see that the AHRS is falling back to DCM which I wouldn’t have expected so I’ll look into that. txs!
Yes, the “moving” GPS is as expected, I specially wanted to do it like this with a poor GPS to show the AHRS fall back to DCM
For @Juergen-Fahlbusch’s log at least the issue appears to be this line in the AHRS code that falls back to DCM if the vertical velocity and position are not available. We should change this so that for ground vehicles, “can_use_ekf” is true even if only the EKF’s attitude is available.
const bool can_use_dcm = dcm.yaw_source_available() || fly_forward;
const bool can_use_ekf = filt_state.flags.attitude && filt_state.flags.vert_vel && filt_state.flags.vert_pos;
if (!can_use_dcm && can_use_ekf) {
// no choice - continue to use EKF
return ret;
} else if (!can_use_ekf) {
// No choice - we have to use DCM
return EKFType::DCM;
}
Note that this fallback only happens if the EK3_SRC1_VELZ/POSZ are configured to use GPS.
TBH, I’m tempted to remove DCM from Rover completely but not everyone on the dev team agrees with me so we might leave that bigger step for another day
Here is a .BIN file from when this happened today.
It took several hours to develop. Note the position jumps in different radial directions (which gradually rotate) and distances.
Thanks very much for the log! This is an interesting log because it’s clear the EKF completely loses its attitude estimate. It’s all fine for a little over 2 hours and then the roll and pitch diverge from reality. The graph below shows the simpler DCM attitude estimate (which is likely correct) vs the EKF estimated attitude.
I suspect this has something to do with GPS-for-yaw being used. I’ll bet that if you set EK3_SRC1_YAW = 3 (GPS with compass fallback) the issue would go away. I’m not saying this is the proper solution of course, just a potential work-around.
I’ll discuss with some of the other devs. txs!
P.S. a replay log would be nice to have bug I also suspect it might be enormous.
Other times I have seen this occur the EKF flags are all “good”. Our companion computer monitors the EKF flags and I have seen this problem (position jumps) happen at the same time as the companion computer thinks, based on EKF flags, that all is well.
Can you tell from the .BIN file whether the EKF flags indicated a problem?
The “solution status” flags make it appear that the EKF thinks it has a position estimate
But looking deeper at the variances it’s clear the EKF is not confident.
Variances over 0.8 will lead to the EKF failsafe triggering and the vehicle stopping (see code here) but again, I’m not saying this is normal or good. Just that we have a check that would stop the vehicle from arming or trying to move autonomously if this happens.
Helpful; I’ll update my companion computer code to look at the variances as well as the flags.
