Had an incident with recent master on a TVBS using EKF3 (using MatekF405-WING so only has EKF3 now) where in flight it became “squirrely” late in the flight in CRUISE and FBWA…so much so that I had to land immediately…
previous builds of master a couple of months ago with EKF3 showed no problems…
logs show that the heading diverged and EKF3 fell back to DCM…but the compass health was perfect and the GPS data seems perfect, but EKF thinks its bad…the next flight it did the same thing…subsequently I tried Stable and Master with EKF2 built and experienced no problems whatsoever…I am concerned that the recent changes to EKF3 (affinitry, lane switches,etc.) might have an issue with single EKF3 builds (1M boards) under certain circumstances, but am not familiar enough with EKF stuff to understand…since compass and gps data was good , there should be no reason for EKF3 to fall back…
here is the log
aside from the question of why there are so many radical GPS innovations when the data looks good, at the very first FBWA segment from QSTAB takeoff, you see the GPS being re-initialized/detected at ~3m 6s while in FBWA…why does this occur???
and it looks like it stopped using compass altogether, even though its healthy and data seems correct, at 3m 10s…???
if I plot the ground track using GPS its perfect and matches planes actual course during the flight, same with AHR2, but POS is jumpy and jagged
What a bumpy ride:
You mentioned above been flown successfully with EKF3 only DEV. builds before. Can you name the version of the older builds (the commit)?
This may help to trace down the issue.
One other detail still bothers me. In your log file I miss a message like:
EKF3 IMU0 origin set
I see both EKF origin set and EKF using GPS missing from gcs messages all the time…was flying this am on a TVBS on EKF2 and waited and waited for the using GPS message, but it never came, but it armed, which it would not have if the EKF and AHRS were not happy…I understand that gcs messages can be lost but I do not know the conditions for that…
as to what version…it was probably about a month ago master…also, I usually add yappu’s bidir stuff so even if I find a log with the commit, it would probably take a while searching back thru gitlog to find a date instead of just searching the master branch directly…
My bad, the “origin set” message seems to be visible only in case of
LOG_DISARMED =1
Found this one: https://github.com/ArduPilot/ardupilot/issues/15341
and your flight
and the symptoms mentioned there look very similar, except the missing EKF’s for obvious reasons.
Maybe you got bitten by this bug?
yep, you right, and I am used to seeing them stream by on my TX so I dont even think about armed/vs/disarmed logging of them…but the issue you reference seems to be one if you armed vertically…this is a TVBS, I arm horizontally in QSTABILIZE then pop the throttle to have it rise vertically and hover, then switch to FBWA to transition to FW…do not think its the exact same issue…Pete has a PR to fix the problem with EKF3 yaw when armed veritcally…should not be the issue with this, I dont think…
your log started with ATT.Pitch = -83.4 degrees.
From my point of view this is close to vertical at least for the IMU.
Here I saw the similarity.
Further I am not familiar with TVBS, just conventional planes.
I still don’t understand the message at the very begin of the log.
2020-10-14 16:07:18 EKF yaw reset -152.91
The start of all evil??
that pitch is correct…its in QSTABILIZE, but laying almost flat on the ground (on a 7deg kick stand)…“level” is vertical in QSTAB… TVBS=twin motor vectored belly sitter…a tailsitter that takes off and lands from horizontal…see this video of it https://youtu.be/0Pw0UojYRfo
What a cool plane.
The video explains very well why you arm/initialize the plane with ATT.Pitch = -83.4 degrees. I already wondered why it started “nose down”.
From point of view of the vehicle attitude it still meets the condition of arming close to vertical.
Maybe arming in a (fixed wing) plane mode like FBWA and switching to QSTABILIZE after arming could avoid the critical condition of arming close to vertical.
I don’t know if there are other constrains against this procedure. It’s just a guess.
I was digging further and found something really concerning.
The status messages at 186 seconds while FLYING at 23.4 meters altitude.
Normally I see this messages BEFORE takeoff only.
A look to the Loop time at this moment shows this:
This indicates a different kind of problems
- bad SD card
- MCU hangups
I can remember a period when logging on the F4 boards was causing lots of trouble
fyi…tried EKF3 again with _MAG_I_GATE doubled to 600 and some minor tweaks to the compass cal from using magfit_WMM utility and had a perfect flight with low innovations throughout…Tridge suspected that the EKF3 was learning bad offsets at the first transition at high current to FBWA and quickly declared the compass innovations too bad and thats why it stopped using compass almost immediately…EKF2 learns the offsets more slowly and had time to recover
Thanks for the update.