POS value (lat/lon) drifting a lot, and crazy synthetic airspeed values

[iforce2d (Chris)]

I set up a new plane recently with 4.2.2, overall it flew OK but I saw a couple of very weird things in the log and on the OSD during the flight. Basically two things:

• the red line in the map (POS.lat and POS.lon) keeps wandering away from the true value and then is corrected suddenly
• the synthetic airspeed value is wildly high at times and grows even while plane is motionless after landing

I’ve uploaded the raw DVR footage from the flight here, where you can see the airspeed values in the OSD. Weird POS and CTUN.As values - YouTube

Normally when I look at a flight log the red and blue lines are so close that they almost look purple, eg.

But when I opened this log the red and blue are quite different. Red is making very sudden movements.

After a while I figured out that the red line is POS.lat and POS.lon. I still don’t know what the blue is, what is that? Here is a section where the POS value jumps a few times:

At first I thought it must be influenced by bad values from some sensor. Here is that same section related to GPS.lat, GPS.alt, BARO.alt and GPS.spd. None seem to show anything unusual.

Here is the same section related to accelerometer values.

And here is the same section related to gyro values.

Here is POS.lat after landing while the plane is not moving.

I noticed that the crazy increase in synthetic airspeed is related to heading. If I understand correctly we can expect some correlation between airspeed and heading, but not so that the airspeed swings this much.

I also noticed that the yaw/heading is slowly changing after landing, while the plane is sitting still, about 100 degrees in 3 minutes. This might be why the airspeed climbs to about 270km/h at that time. Looks a bit silly on the OSD (see the video at 29:50). The gyro values seem healthy, I see no reason for the yaw to move so much while motionless.

In all my planes until now I have been using 3.9.8 so this is the first time I’ve tried anything newer - is there some new setup requirement I might have missed? There is no compass onboard.

So far I’ve mainly only used manual and FBWA modes so this weirdness wouldn’t have mattered, but going forward I’m hoping to try all the usual autonomous stuff too. I’m just a little uneasy about doing that without knowing a bit more about what’s going on here. Any tips would be appreciated.

ArduPlane V4.2.2 (693c9d91)
ChibiOS: 93e6e03d
omnibusf4pro 00310046 38395006 20373654
RCOut: NeoP:1-2 PWM:3-5 NeoP:6 PWM:7-8
IMU0: fast sampling enabled 8.0kHz/1.0kHz

[iforce2d (Chris)]

hmmm… ok. I did another flight today without changing any settings, and everything was fine this time.

[iforce2d (Chris)]

I flew this plane again today and unfortunately the problems are back with a vengeance.

This time in addition to the speed values going crazy, the altitude was too, so I did not dare try RTL. For about 15 minutes I was able to fly ok in FBWA until suddenly I saw “DCM active” / “EKF3 active” alternating. When in EKF3 the artificial horizon was spinning around and the plane tried to follow suit. If I did not have FPV video and I was not watching it at the time, I probably would have lost the plane. For the next 20 minutes the artificial horizon kept spinning, even after landing.

Here is a clip of the problem moment after which I had to use manual mode, and another showing values going crazy even after landing:
https://youtu.be/4Dy6xvB69Ic

The log is again showing discrepancy between blue and red lines, this time much worse and pretty much throughout the entire flight. In this screenshot the further out section toward the right is where the above clip happened, right where a particularly large diversion between red and blue occurred:

Not sure if it’s related, but while in FBWA for the first part of the flight, there were small wobbles in roll occurring rhythmically about every 2 seconds, these were only noticeable in the HD footage afterward:

rollWobbleProblem

Please PM me if you’d like to see the dataflash log. At first I suspected bad sensor data but I can’t see any (checked GPS lat/lon/alt, BARO alt, IMU gx,gy,gz,ax,ay,az, anything else that would mess up EKF?).

I’d like to go back to 3.9.8 but I need to modify some options in the hwdef for my setup (current sense on RSSI pin), so I can’t use the official build as is. Does anyone know how I should reset my git checkout so that it’s at Plane 3.9.8 ?

[iforce2d (Chris)]

Here is a screenshot of the aileron output during the rhythmic wobbling shown in the video above. It appears to be related to the divergence between red and blue lines on the map view.

I eventually figured out how to build 3.9.8 with my hwdef change, so I’ll try that next and see if there’s any difference. It will not be for a while though because today’s landing ripped a motor off - testing 2.2kg payload resulted in a heavy landing in rough grass

[iforce2d (Chris)]

I was looking at the log of the third flight a bit more today using the online viewer, which is a bit easier to navigate and set up expressions.

There appears to be vibration in the accelerometer Y value, I’m not sure how significant this is to the EKF but it’s causing some jello in the HD video. Here is the moment where it switched into DCM briefly just before going berserk.

The gyro values at that time look clean.

There is something called VIBE, not sure how it is calculated but the Y axis again shows higher values.

Regarding the ‘rhythmic wobbles’ mentioned above, here is the gyroX and aileron together. They appear to be moving at exactly the same time which I would interpret as the gyro is causing the aileron. If the aileron output was causing the gyro movement I would expect at least a slight delay as the servo moves, air flow has to change etc.

The rhythmic wobbles were at a time in the flight when the accelerometer seemed to be less subject to vibration (probably lower motor throttle). Here are all the IMU values of a single ‘wobble’ event.

Only gyroX looks really out of place. Could a faulty gyro cause sudden oscillations like this maybe? I wonder why there were no problems at all in the second flight I did.

Here are some top-down screenshots of each flight.
Flight 1 seemed to hold things together relatively well, I could use RTL.

Flight 2 was basically flawless.

Flight 3 was chaos for most of the flight.

Side view of flight 2

Side view of flight 3

[Allister (Allister Schreiber)]

Have a look at the GPS accuracy. (GPA.HACC, GPA.VACC, and GPA.SACC). HACC and VACC can be higher (2 or 3), but SACC should well below 1, around .5 or better.

If those values don’t look favorable you could try GPS_GNSS_MODE,5 or 65. This will turn “down” the GPS module for what systems it’s following and possibly reduce the working load.

[iforce2d (Chris)]

Here is GPA.Hacc, GPA.Vacc, GPA.Sacc over the whole flight. Interestingly they are all quite low just before the major problem point (21:20 in the timeline, where I changed to manual mode).

Here is a bit closer up on the problem moment.

One slightly unusual thing I’ve noticed about this build is the rather low number of satellites reported by the GPS. In the past I’ve seen these modules showing over 20 satellites quite commonly. But with this build only 15 at the most. Seems like maybe it’s already only following one system.

[Cightline (Sightline)]

Could it be solar activity?

Some (old) info here: Mission Planner Solar Storm Warning - #3 by dcwalmsley

[Liviu_Grigorescu]

The IMU.gyro samples look like the IMU sends bad data. Your values oscillate a lot around 0 which seem random to me.

To compare, here is one of my flights where I zoomed in on gyroy. Even if it oscillates, it is not random and it follows the general plane movement.

You can check vibrations from main Mission Planner dashboard.

Generally the values have to be below 30. They have to be checked throughout the whole throttle range not only at max.

If it is not the vibrations I would suspect bad IMU/hardware.

[Allister (Allister Schreiber)]

Those values look really high to me. Try the setting I mentioned above.

Have you run MagFit on this plane?

[VRquaeler (sascha)]

There is no compass on this plane.
You can see in the first vid above the “typical EKF DCM lane switching” known for a (bad) EKF3 without compass sensor.

[VRquaeler (sascha)]

@iforce2d

Here is GPA.Hacc, GPA.Vacc, GPA.Sacc over the whole flight. Interestingly they are all quite low

Hi Chris, from my understanding this is NOT a low value. Even with a cheapo BN220 my builds stay GPA.Sacc below 0.5 and average at 0.2.

The lower than usual Sat-count mentioned by you earlier may be a symptom of a underlying GPS or wiring problem.

[iforce2d (Chris)]

@Liviu_Grigorescu I was doing an autotune of roll at that time, so I think those gyro values are actually ok. You can see similar oscillations in the Acc, Alt as well. Sorry, forgot to mention it.

@Allister @VRquaeler Actually I don’t know much about those values other than the expected norms Allister mentioned. I was just meaning they went relatively low just before the problem moment. When I checked the log of flight 2 (the flawless one) these values look about the same. The low sat count could be due to having the GPS tucked down beside the wing spar, not the best location.

Today I was looking at some values from the EKF3 section of the log, so here’s another deluge of screenshots In all following screenshots I have marked the problem time (21:19.5) with black vertical line.

Gyro biases start to change about 1.5 seconds before the problem, with a sudden jump about 0.5 seconds before.

Accel bias

Gyro sensor activity at that time, not much happening until after the black line.

Likewise for accel sensor

EKF solution status drops to about 40k around 2 seconds before, then jumps back to 120k or so at the problem moment. During that low time I saw “DCM active”

EKF error score

Here is an overall view of solution status for the entire flight.

EKF Pos NE

EKF position innovation NE

EKF velocity innovation NE

EKF euler roll (I started a long turn at 21:00 so this looks ok)

EKF euler pitch. Pitch felt a bit sloppy and not as responsive as I’m used to with FBWA.

EKF euler roll vs gyro sensor (the two EKF values are on top of each other)

Here is a clip of the OSD view covering the same timespan as that last graph.

ekf3ProblemOSD

The sequence of events (as marked timed by the on-screen timer) were:

15:52 a long slow climb (~2 minutes) done by holding pitch stick back a little in FBWA
16:02 let go of pitch stick, plane initially returns to level
16:06 plane now wants to nose up to about the same angle as the climb was
16:10 I push pitch stick forward to keep level, and start a right turn. During the turn I have to keep pitch pushed forward a tiny bit to keep level.
16:24 from here until 16:32 I my fingers were perfectly motionless, but the plane wants to nose up a bit more at 16:28. I see “DCM active” for about two seconds.
16:32 problem moment

Regarding the rhythmic roll wobbles I mentioned earlier, there is a very tight adherence to a 2 second period. Is that likely to be caused by the sensor itself for a while, and then come right for the rest of the flight? Seems unlikely. The updates of the EKF seem to be based on a period that is near to 2 seconds, but can vary? In this graph the timing of roll wobbles and the EKF position innovation jumps appear to match up, but on closer inspection they don’t really. Still it kinda looks noteworthy.

fwiw here are the full flight graphs of EKF position innovation NE

… and EKF velocity innovation NE

[timtuxworth (Tim Tuxworth aka TimthePlaneman)]

This is a bit of a “what came first” - but I noticed the altitude problems first so went down that rabbit hole. This is what the EKF Pos D (D is down I think) looks like at the same time.

But whichever way you look at it, it seems like “something happened” - right at that critical time. Did your payload shift perhaps?

Oh and that does seem to show that the Altitude problem happened first. This seems to be supported by this, which shows the baro going off right about the same time.

[timtuxworth (Tim Tuxworth aka TimthePlaneman)]

Oh this is fun! Using the UAV Log Viewer, I captured this showing what ArduPlane thinks the plane is doing. It’s crazy!

Screen Recording 2023 01 24 at 9 54 16 AM

[VRquaeler (sascha)]

@iforce2d

Gyro biases start to change about 1.5 seconds before the problem, with a sudden jump about 0.5 seconds before.

Hi Chris,

Glad to see You discovered the suspicious Gyro bias by yourself. Found a few other things. More tio come.

[VRquaeler (sascha)]

@timtuxworth

I captured this showing what ArduPlane thinks the plane is doing. It’s crazy!

Hi @timtuxworth

That’s what you can see if the EKF goes mad. Good catch.

[iforce2d (Chris)]

@timtuxworth Yes the calculated altitude drops from 860m to 165m when the DCM becomes active, I noticed that on the OSD while I was flying. The plane was doing a very slow calm turn at the time so I doubt the payload moved (it was still in place after landing) or that payload movement would somehow result in an instantaneous 700 meter altitude change.

The sudden altitude change sensed by the baro in your third graph is when I was frantically trying to prevent a crash, it could have easily descended 30-40 meters in a few seconds. If you zoom in you should see that it happens slightly after the 700m jump in EKF Pos D.

@VRquaeler What can be done about it though? If I understand correctly the EKF decided that the accel/gyro offsets it was using needed to be tweaked, so it just overwrites them with values calculated from other sensors? Seems a bit risky. I wonder if the long slow steady shallow turn I was doing has any relevance. In a non-EKF system a turn like that can make a simple complementary filter lose track of real gravity direction.

[timtuxworth (Tim Tuxworth aka TimthePlaneman)]

More than that @iforce2d - notice the crazy attitude changes. The plane is (logically) spinning in space, pointing up, down, backwards, forwards, left, right … I’m honestly puzzled. Looking forward to what @VRquaeler has to add.

[VRquaeler (sascha)]

Hi,

Looking closer to the log files I found a distinct difference between the good (2nd flight) an the bad (3rd) flights.

The roll-pitch comparison between DCM and EKF for the good flight matches fine.

The same comparison for the “mad” flight show’s a significant difference from the very begin of the flight, shown below.

It may be part to make the EKF unhappy.

Comparing the parameters of the different flight’s I found a small change in the AHRS alignment.

I saw small changes in the gyro offset’s. They are small and should be fine… The gyro offsets are created while the flight controller is initializing.

Now we have to understand why this happens and how to avoid a mad EKF.

I have a few questions left.

Did you change the yaw alignment of the flight controller ?
Did you (unintentionally) move/rotate the plane while initiating for the third flight?