POS value (lat/lon) drifting a lot, and crazy synthetic airspeed values

I was looking at the log of the third flight a bit more today using the online viewer, which is a bit easier to navigate and set up expressions.

There appears to be vibration in the accelerometer Y value, I’m not sure how significant this is to the EKF but it’s causing some jello in the HD video. Here is the moment where it switched into DCM briefly just before going berserk.

The gyro values at that time look clean.

There is something called VIBE, not sure how it is calculated but the Y axis again shows higher values.

Regarding the ‘rhythmic wobbles’ mentioned above, here is the gyroX and aileron together. They appear to be moving at exactly the same time which I would interpret as the gyro is causing the aileron. If the aileron output was causing the gyro movement I would expect at least a slight delay as the servo moves, air flow has to change etc.

The rhythmic wobbles were at a time in the flight when the accelerometer seemed to be less subject to vibration (probably lower motor throttle). Here are all the IMU values of a single ‘wobble’ event.


Only gyroX looks really out of place. Could a faulty gyro cause sudden oscillations like this maybe? I wonder why there were no problems at all in the second flight I did.

Here are some top-down screenshots of each flight.
Flight 1 seemed to hold things together relatively well, I could use RTL.

Flight 2 was basically flawless.

Flight 3 was chaos for most of the flight.

Side view of flight 2

Side view of flight 3

Have a look at the GPS accuracy. (GPA.HACC, GPA.VACC, and GPA.SACC). HACC and VACC can be higher (2 or 3), but SACC should well below 1, around .5 or better.

If those values don’t look favorable you could try GPS_GNSS_MODE,5 or 65. This will turn “down” the GPS module for what systems it’s following and possibly reduce the working load.

Here is GPA.Hacc, GPA.Vacc, GPA.Sacc over the whole flight. Interestingly they are all quite low just before the major problem point (21:20 in the timeline, where I changed to manual mode).

Here is a bit closer up on the problem moment.

One slightly unusual thing I’ve noticed about this build is the rather low number of satellites reported by the GPS. In the past I’ve seen these modules showing over 20 satellites quite commonly. But with this build only 15 at the most. Seems like maybe it’s already only following one system.

Could it be solar activity?

Some (old) info here: Mission Planner Solar Storm Warning - #3 by dcwalmsley

The IMU.gyro samples look like the IMU sends bad data. Your values oscillate a lot around 0 which seem random to me.

To compare, here is one of my flights where I zoomed in on gyroy. Even if it oscillates, it is not random and it follows the general plane movement.

You can check vibrations from main Mission Planner dashboard.


Generally the values have to be below 30. They have to be checked throughout the whole throttle range not only at max.

If it is not the vibrations I would suspect bad IMU/hardware.

Those values look really high to me. Try the setting I mentioned above.

Have you run MagFit on this plane?

1 Like

There is no compass on this plane.
You can see in the first vid above the “typical EKF DCM lane switching” known for a (bad) EKF3 without compass sensor.

1 Like

Hi Chris, from my understanding this is NOT a low value. Even with a cheapo BN220 my builds stay GPA.Sacc below 0.5 and average at 0.2.

The lower than usual Sat-count mentioned by you earlier may be a symptom of a underlying GPS or wiring problem.

@Liviu_Grigorescu I was doing an autotune of roll at that time, so I think those gyro values are actually ok. You can see similar oscillations in the Acc, Alt as well. Sorry, forgot to mention it.

@Allister @VRquaeler Actually I don’t know much about those values other than the expected norms Allister mentioned. I was just meaning they went relatively low just before the problem moment. When I checked the log of flight 2 (the flawless one) these values look about the same. The low sat count could be due to having the GPS tucked down beside the wing spar, not the best location.

Today I was looking at some values from the EKF3 section of the log, so here’s another deluge of screenshots :slight_smile: In all following screenshots I have marked the problem time (21:19.5) with black vertical line.

Gyro biases start to change about 1.5 seconds before the problem, with a sudden jump about 0.5 seconds before.

Accel bias

Gyro sensor activity at that time, not much happening until after the black line.

Likewise for accel sensor

EKF solution status drops to about 40k around 2 seconds before, then jumps back to 120k or so at the problem moment. During that low time I saw “DCM active”

EKF error score

Here is an overall view of solution status for the entire flight.

EKF Pos NE

EKF position innovation NE

EKF velocity innovation NE

EKF euler roll (I started a long turn at 21:00 so this looks ok)

EKF euler pitch. Pitch felt a bit sloppy and not as responsive as I’m used to with FBWA.

EKF euler roll vs gyro sensor (the two EKF values are on top of each other)

Here is a clip of the OSD view covering the same timespan as that last graph.

The sequence of events (as marked timed by the on-screen timer) were:

15:52 a long slow climb (~2 minutes) done by holding pitch stick back a little in FBWA
16:02 let go of pitch stick, plane initially returns to level
16:06 plane now wants to nose up to about the same angle as the climb was
16:10 I push pitch stick forward to keep level, and start a right turn. During the turn I have to keep pitch pushed forward a tiny bit to keep level.
16:24 from here until 16:32 I my fingers were perfectly motionless, but the plane wants to nose up a bit more at 16:28. I see “DCM active” for about two seconds.
16:32 problem moment

Regarding the rhythmic roll wobbles I mentioned earlier, there is a very tight adherence to a 2 second period. Is that likely to be caused by the sensor itself for a while, and then come right for the rest of the flight? Seems unlikely. The updates of the EKF seem to be based on a period that is near to 2 seconds, but can vary? In this graph the timing of roll wobbles and the EKF position innovation jumps appear to match up, but on closer inspection they don’t really. Still it kinda looks noteworthy.

fwiw here are the full flight graphs of EKF position innovation NE

… and EKF velocity innovation NE

This is a bit of a “what came first” - but I noticed the altitude problems first so went down that rabbit hole. This is what the EKF Pos D (D is down I think) looks like at the same time.

But whichever way you look at it, it seems like “something happened” - right at that critical time. Did your payload shift perhaps?

Oh and that does seem to show that the Altitude problem happened first. This seems to be supported by this, which shows the baro going off right about the same time.

Oh this is fun! Using the UAV Log Viewer, I captured this showing what ArduPlane thinks the plane is doing. It’s crazy!

Hi Chris,

Glad to see You discovered the suspicious Gyro bias by yourself. Found a few other things. More tio come.

Hi @timtuxworth

That’s what you can see if the EKF goes mad. Good catch.

@timtuxworth Yes the calculated altitude drops from 860m to 165m when the DCM becomes active, I noticed that on the OSD while I was flying. The plane was doing a very slow calm turn at the time so I doubt the payload moved (it was still in place after landing) or that payload movement would somehow result in an instantaneous 700 meter altitude change.

The sudden altitude change sensed by the baro in your third graph is when I was frantically trying to prevent a crash, it could have easily descended 30-40 meters in a few seconds. If you zoom in you should see that it happens slightly after the 700m jump in EKF Pos D.

@VRquaeler What can be done about it though? If I understand correctly the EKF decided that the accel/gyro offsets it was using needed to be tweaked, so it just overwrites them with values calculated from other sensors? Seems a bit risky. I wonder if the long slow steady shallow turn I was doing has any relevance. In a non-EKF system a turn like that can make a simple complementary filter lose track of real gravity direction.

More than that @iforce2d - notice the crazy attitude changes. The plane is (logically) spinning in space, pointing up, down, backwards, forwards, left, right … I’m honestly puzzled. Looking forward to what @VRquaeler has to add.

Hi,

Looking closer to the log files I found a distinct difference between the good (2nd flight) an the bad (3rd) flights.

The roll-pitch comparison between DCM and EKF for the good flight matches fine.

The same comparison for the “mad” flight show’s a significant difference from the very begin of the flight, shown below.

It may be part to make the EKF unhappy.

Comparing the parameters of the different flight’s I found a small change in the AHRS alignment.

I saw small changes in the gyro offset’s. They are small and should be fine… The gyro offsets are created while the flight controller is initializing.

Now we have to understand why this happens and how to avoid a mad EKF.

I have a few questions left.

Did you change the yaw alignment of the flight controller ?
Did you (unintentionally) move/rotate the plane while initiating for the third flight?

@VRquaeler the small changes in AHRS trim were done by me. In the first flight the plane seemed to nose down a little more than was ideal for cruising in FWBA, so I trimmed it up about 1 degree iirc, and tweaked the roll trim too.

I didn’t change the yaw alignment. The plane was pretty much motionless during the initialization stage. It launches from a dolly so it wasn’t even being moved around like it would be for a hand launch.

One unusual thing about this build is that the flight controller is mounted on the wall of the electronics area, instead of on the floor like I would usually do. The AHRS_ORIENTATION parameter is set to 22 which is “Yaw90Roll270”. This gave the correct responses in the HUD and for the control surfaces, and managed two flights with no strange behavior, but looking at it now I can’t picture how “Yaw90Roll270” fits with the mounting I have, unless these movements are backward to what I was thinking. That is to say, if I took the FC from the default orientation and rolled it 270 first, and then yawed it 90 it would match the mounted position. I couldn’t find any docs about how we’re supposed to interpret these descriptions.

Looking at the OSD footage from the good and bad flights, I notice that shortly after launch when I see the “EKF3 IMU0 yaw aligned” message, the heading jumps by only 3 degrees in the good flight, and in the bad flight it jumps by about 70 degrees. In the bad flight it returns to the same value it had before takeoff.

Here are clips of the pre-launch section for each flight.

Good flight:

Bad flight:

For reference, the concrete airstrip runs at about 70 degrees, so my direction of launch is about 65 degrees.

In both cases, the heading starts off near the default heading of north (eg. 356 degrees) after powerup and doesn’t change much until the plane is armed, at which point it shows “DCM active”, the heading jumps to about 260 and starts to slowly drift a little. I think this is probably normal right?
As soon as the takeoff roll starts, the heading jumps to the correct value of about 66 degrees, and continues to look correct for a while after lifting off.
After about 15 seconds the message “EKF3 IMU0 yaw aligned” appears and the heading changes again. In the good flight it jumps from 10 degrees to 7 degrees. In the bad flight it jumps from 64 to 356. In the other “not quite as bad” bad flight it jumped from 46 to 99 degrees.

In the good flight, the plane happened to be heading very close to north at the time of the first EKF alignment, so even if the value it jumped to was poorly calculated, it would still have been close to the correct value just by lucky chance. Maybe that helped make the good flight a success.

2 Likes

Hi Chris,

Have done very similar things ion the past. In general it’s not the issue as long as the changes are small, like in your case.

Just asking. I have flown with vertical oriented FC’s too. Regarding the log files this is not the issue. The accelero meter calibration values look OK.

Assuming the plane is your new “Twin Hopper” it’s unlikely to move the plane unintended.

Very good catch, I guess we come closer to the root cause.
A bad yaw alignment is sufficient to explain all the EKF madness.

By accident I discovered the yaw orientation of the plane while initializing can make a BIG difference. A plane without compass start to believe pointing to North after the initializing.
For planes (non VTOL fixed wing planes) without a compass I put them down pointing North before booting up. In general I get way more reliable flights keeping this procedure.

The other alternative is to add a compass.

For me the living in the Netherlands it’s often windy and the flight performance improvement especially in cross wind conditions was worth all the effort to have a healthy compass.

The compass has to be put in a place with lowest possible interference of power wires, magnets etc. The compass has to be calibrated properly.

Flying with a compass is still NO guaranty against all the DCM/EKF switching, especially in case of a bad (calibrated) compass.

To avoid the weird moments a compass check is part of the preflight checks. The yaw alignment of the plane and the Mag Field has to be within a expected and physical plausible range. On Mission Planner I have added this on the “Quick tab” see pic below.

There are other alternatives like flying with DCM only. Today I would not recommend this anymore.

PS: Keep going with your creative video’s. I got a lot of inspiration and new ideas watching your channel. Thank you!!!

Not sure if its related or not but I had a crash which destroyed my plane where I lost control and my YAW jumps to crazy value after "“EKF3 IMU0 yaw aligned” message.

Here is my thread: EKF3 Yaw alignment on V4.4 dev brought my plane down

I switched back to Plane 3.9.8 and had 4 trouble-free flights so far. So I think I will just stick with that, it does everything I need.

1 Like