I have been testing 3.4 recently and was checking some motor imbalance issues in ALTHOLD and POSHOLD. This is a newish machine and I have not begun to tune it so most of the settings are defaults. Even s,o it is very stable and flies quite well, with room for improvement but I am very happy with it so far.
On this particular flight I had two GPS units attached, even though I know I shouldn’t just to see what the relative differences were. I was actually working on a motor imbalance where I am seeing motors 1&2 constantly higher than 3&4. Strangely it flies straight and throttle inputs do not result in any noticeable yaw. I think there is a twist somewhere but cannot find it.
In general both ALTHOLD and POSHOLD are working very well. Very little drift and no strange yaw or altitude oscillations. It is like it is on rails most of the time.
About 8 minutes into the flight the active GPS glitches and changed position radically by about 50m from my backyard all the way to the street. It took about 40 seconds for the GPS position to come back into line with the actual position of the quad.
When the glitch happened I was in POSHOLD mode, hovering, with no stick inputs. I saw the quad jump a bit and corrected. I had to correct a couple of times before I felt the need to switch to ALTHOLD.
Overall the whole thing was quite tame and at no time did I feel as if the quad was out of control. It did not suddenly fly off into a wall. I am not sure if the EKF recognised the glitch and I whether having two GPS units added to or helped with the problem. I know that two GPS’s is not recommended but I was curious to see what would happen. I have gone back to the single M8N.
I can see in the logs that the desired position diverges greatly from the actual position but this did not seem to translate into a sudden uncontrollable change of attitude. It did move away but I could easily correct it and take control. This may be due to the max accelerations in auto being set quite low.
I am interested to find out how the EKF handled this and what it did to keep the craft stable in POSHOLD when the GPS location changed. I did manually correct but did not have to radically compensate.
I would appreciate any insight into how the firmware handled this.
Just a side note, I don’t think there was any EKF failsafe or error detected here. Should this have been handled as a failsafe event?
I guess my question is, did the EKF simply handle the position error internally and reject bad GPS data in favour of predicted positions based on IMU data?
OR
Should this have caused a Failsafe state?
I don’t know enough to read the log in this detail and any help would be appreciated in helping me understand this.
Thanks
It seems GPS1 was used all the time. But GPS1 Status switched between 3 and 4 several times. This is problematic. What is also interesting is that GPS1 lost about 6 Sats in a short period of time. This might be related.
I once had a small glitch of some meters some seconds after takeoff induced by SBAS kicking in. This can be very dangerous especially in that period of flight. Since then I switched SBAS off.
At the moment the GPS glitch happened it had switched to GPS2. Like @Thorsten said your GPS1 status was changing between 3D and DGPS, but, in a moment it was in 3D, your GPS2 changed to DGPS, and so it was chosen to be used. Unfortunately at that time the glitch happened.
Regarding your question about failsafe: the glitch was long, but in the middle of it, the GPS1 started to be used again, so to the EKF it lasted about 13 seconds. The EKF did detect the GPS was having a glitch, but I don’t think we are doing anything with it and we probably should - but I’m far from being an EKF expert. The position innovations only went really high when it changed back to GPS1 - for about 7 seconds. I don’t know how the EKF managed to have such high innovations for so long without getting upset, but it apparently did.
I’ll talk with @priseborough about this to see if we can improve anything.
Thanks @Thorsten and @OXINARF for the replies. That helps me make much more sense out of what happened. I have removed the second GPS and will make sure that SBAS is off and see if that improves things.
As I mentioned, this is a new build and most of the parameters should be defaults .
I am actually quite pleased that the firmware behaved as it did as it did not result in a crash. I saw the high innovations when I looked at the log and it surprised me that the problem was so long. Normally the innovations are very low and consistent. Maybe I have slow reflexes but it only felt like a small jump and a couple of corrections before it all became stable again. It might have been seven seconds but it felt like less. I didn’t even feel the need to go back to Stabilise immediately. (I had my finger on the switch though!!)
If the EKF did become upset it seems that it recovered as it should have (or at the very least it kept together enough to keep flying without trying to take off in some random direction). I seem to recall seeing somewhere in the code a few changes to how the EKF recovers from bad sensor data and resets. It would be great to have Paul have a look and I hope that the log is of use.
Once again, thank you both for the assistance it is very much appreciated.
Maybe this comment should be splitted in somewhere, sorry if I didn’t chosen the right place.
I’m testing beta v3.4 and want to comment that when I try to arm before having enough GPS satellites, it won’t arm when have enough already. I have to wait to arm after having proper GPS signal to avoid this bug.
With this message I just want to report the glitch.
Awesome work which you are doing with this Ardupilot version!
The bug is that if it is ordered to arm before having proper satellite connection, it won’t arm when it have it. Now, it is compulsory to wait for arming until having GPS connection or it will be impossible to arm and fly when the GPS link is good.
Because our switching between GPS is really basic and it doesn’t handle the jump between GPS positions. You can use it, but you should be aware of the issues with it.
Ah, gotcha. I’ve had it switch mid flight before, it veered about a meter from where it was hovering, then went back (I think it went back to the original GPS). Isn’t a slight twitch better than a complete loss of position?
Yes, but like I said the issue is that the current switching algorithm is very simple: it only checks for fix status and number of satellites. That means that if, for example, you mix a GNSS system with just GPS and one with GPS and Glonass the system can actually switch to the second one because it has more satellites, even if the position accuracy is actually worse.
So, it is generally not recommended to run different GPS types. Similar types should be less prone to problems, but can still be affected.
Gotcha. I added the second GPS since I was going a lot of follow mode and missions, I rarely fly manually in GPS assisted modes. Would it be recommended that I remove it?
If you are using different GPS types, yes, I don’t think that our current code is sufficiently good - and you are using it in autonomous modes which can lead to something bad if you don’t have your controller in your hand and react rapidly.
They are both 3DR ublox NEO-7 (the one built into the Iris+ and a second one on a mast). I will pop it off anyways, it sounds like there are more disadvantages than advantages.
Thanks for your help/input.