Servers by jDrones

Loss Of Altitude in Auto with Copter 3.5

(Chris Olson) #1

This is not a problem unique to helicopter. Multi-rotor pilots have noted this as well but it’s usually blamed on the aircraft not being able to maintain altitude due to running out of power and dismissed. With helicopters we don’t have that problem so I can more accurately illustrate what is going on. I noted this during testing and tuning the WPNAV_ACCEL on this heli and dismissed it at the time as a one-time anomoly. But it turns out it’s not.

This is a screenshot summary of the flight plan - it is flown at 15 m/s. The Do Jump command sends us back to waypoint 2 seven times so the heli flies a high-speed figure-8 on a fairly small course with tight turns.

New photo by Chris & Kristin Olson

This the course flown with AC3.3.3 based software, looking at barometer altitude vs GPS altitude. There is no problems other than GPS altitude is notoriously inaccurate. This older software does not show the mission commands on the timeline but when the heli comes into the final waypoint it is at the proper altitude of 8 meters. Home and the final waypoint are one and the same.

This is the same course flown with the same helicopter, same hardware, same speed - only thing different is that the Pixhawk is loaded with AC3.5.3. Notice the steady loss of altitude measured by the barometer (and verified both visually and with onboard camera reference). At the final waypoint instead of being at 8 meters AGL (flight planned) the heli comes in at only 1.3 meters AGL. While GPS still thinks it’s at 8 meters above ground (using height above the reference ellipsoid used by GPS). The GPS altitude jumps all over the place like it always does. The barometer tells the real story of what’s going on. At landing the ground station indicates the heli is at 6 meters altitude when it’s sitting on the ground.

Here’s what’s strange about this. If I fly a long mission at 30 m/s that has few turns it maintains altitude fine. But if the heli is continuously making high-G turns and high-speed passes like on this test course, or a survey flight, the actual altitude gradually decays. I had to abort a survey flight today flown with AC3.5.3 when it became evident in the FPV feed that the heli was getting dangerously close to tree tops in the turns on the end of the passes. I didn’t trust its altitude estimations at all so I flew it 1.6 km home on FPV manually, and landed it manually. At landing the ground station said it was at 17 meters altitude, landing on the same spot it took off from. The barometer showed a momentary dip to -1.2 meters (caused by pressure buildup under the main rotor during landing close to the ground), then came back up to .5 meters. So where is the ground station getting the 17 meters from? Which is obviously what the autopilot thinks the heli is at sitting on the ground. And it obviously thought it was 17 meters higher than it really was in flight.

I loaded the helicopter in the truck, went back home and got my other heli with the AC3.3.3 based firmware in it and flew the survey with that one without any issues.

I don’t know exactly what’s causing this. But I would suggest a flaw in the altitude estimation in the 3.5 firmware when flying at high speed, changing heading a lot, and flying spline waypoints. The weird thing is that in the log of this test flight, at the final waypoint (flight planned at 8 meters) the CTUN desired altitude shows 1.3 meters, which agrees with the barometer and what it was actually at at that waypoint, but this does not agree with the flight plan.

So at this point, I still cannot fly the 3.5 firmware on commercial flights until I figure out what the problem is.

20m altitude difference POS ALT vs GPS ALT
Pixhawks 3.5.3 AltHold Crash. Pos(Vert) issue?
(Jakob Schmidt) #2

I saw same issue on my last flight. Further, I saw it ‘jump’ when I switched from stabilize to loiter.

(Chris Olson) #3

I’m fairly convinced it is a flaw in whatever the firmware is using for altitude estimation, which may be tied to the EKF estimation at arming. Multi-rotor pilots have noted it as well.

For now I’ll continue to fly my custom fork in my commercial heli’s, as when I come in for landing it’s kind of nice to see the altitude at zero when the skids touch the ground. And I’ll continue testing 3.5 under more controlled conditions to see if I can nail down the problem to submit a proper issue on it on GitHub.

(Jakob Schmidt) #4

Got back on my PC. Was supposed to run the whole mission at 4 meters and it very clearly wasn’t and I had to take manual control, as it got real close to a small tree (2 meter tall)

(Chris Olson) #5

The altitude estimation is obviously drifting and it appears to be a gradual drift. It’s not related to barometer drift, as that’s the only instrument on the aircraft that indicates accurately what’s going on. I think in 3.3.3 the software uses the barometer for altitude in auto. I don’t know about 3.4 as I never tested that enough. But in 3.5 it’s using something else because if it was using the barometer it wouldn’t be doing this. For me it only happens if the helicopter is running faster speed >15m/s and making a lot of turns.

Looking my logs over it looks like it’s placing more weight on the GPS estimation of altitude instead of the barometric pressure. Which would be a disaster if it’s really doing that because GPS altitude can’t be used in even high-end certified units in full-size aircraft.

I’m not going to get a chance to look at the issue more right away as I have a lot of flights to make this week. And I need both helicopters so I’m reverting this one to the 3.3.3-based firmware this morning so I can use it. I don’t know much (as in basically nothing) about the EKF code and how the navigation system works. But when I get a chance I’ll make some more test flights under controlled conditions where it’s easier to recover with manual control. And see if I can figure out what’s causing it. Coming in for landing with the altimeter indicating 17 meters is DEFINITELY a serious problem, which would be consistent with using GPS for altitude, which in my experience using it in full-size aircraft is only accurate maybe +/- 30 meters.

(Jakob Schmidt) #6

There’s a parameter for GPS/Barometer weighing…I’ll dig it up later, when I get on a computer again.

(Chris Olson) #7

It’s EK2_ALT_SOURCE which I have set to zero and it’s supposed to use the barometer. But obviously it’s not, because if it was it wouldn’t do what it does. The barometer readings in the logs are spot-on, agreeing with what’s actually happening. And the flight plan is not using the altitude information from the barometer, because if it was the helicopter wouldn’t be tending to run into trees that it’s supposed to be 60 feet over the top of.

There’s some other adjustments for there for the filter that reduces or increases weighting of the baro measurement, etc… Whatever is going on there does not work. Or the filter doesn’t work properly. Or it’s applying some other filter, or fusing of GPS, regardless of the EK2_ALT_SOURCE setting. Or something. Whatever it is, it is not safe to fly on an serious autonomous flight.

(Jakob Schmidt) #8

I could have sworn that there was a different parameter with an actual weighing value, but I cannot find it.

Changes in barometric pressure (and temperature variations influencing the onboard sensors on the pixhawks) is a fairly common issue on fixed wings, which can make auto-landings after longer missions an issue, but then we’re talking 1-2 meters, not 17!!.

(Chris Olson) #9

Correct. I don’t get too excited about 1-2 meters because pressure buildup under the main rotor can cause that. But 7 meters off on a short flight, and 17 meters off on a long one - there’s something wrong with the altitude estimator. And the setting of zero for the alt source is obviously not limiting the altitude estimation to just the barometer. It’s being fused with some other sensor readings, and I suspect it’s GPS.

(Chris Olson) #10

According the documentation:

This parameter controls which height sensor is used by the EKF. If the selected optionn cannot be used, it will default to Baro as the primary height source. Setting 0 will use the baro altitude at all times. Setting 1 uses the range finder and is only available in combination with optical flow navigation (EK2_GPS_TYPE = 3). Setting 2 uses GPS. When height sources other than Baro are in use, the offset between the Baro height and EKF height estimate is continually updated. If a switch to Baro height needs to be made when the filter is operating, then the Baro height is corrected for the learned offset to prevent a sudden step in height estimate.

If setting EK2_ALT_SOURCE to zero is using the Baro altitude, why is the helicopter not at the correct altitude when the Baro obviously shows it’s not at the correct altitude? Taking a snapshot of this one waypoint the command altitude value (flight plan) was 7.62 meters, which is 25 feet. The helicopter was actually at 1.5 meters, or 5 feet (my visual estimation was 1 meter, or about 3 feet).

Further, since the command altitude was 7.62 meters, why is the CTUN.DAlt only 1.58 meters? How can you have a desired relative altitude of 1.58 meters at that point and space in time when the flight plan is plainly 7.62 meters? Doesn’t make sense. Isn’t the flight plan command altitude the desired altitude?

Tracing the GPS altitude estimate from the beginning of the flight to the end, the GPS estimate (height above the reference ellipsoid) agrees with the flight plan. None of the rest does except right at the beginning of the flight, and it went downhill from there (pardun the pun). So my conclusion is that it used GPS altitude for the entire flight and totally ignored the barometer. In other words, what the documentation says it’s supposed to do vs what it actually does in the real world is two different things. All the EKF2 settings are default:

EK2_ABIAS_P_NSE , 0.005
EK2_ACC_P_NSE , 0.6
EK2_BCN_I_GTE , 500
EK2_EAS_I_GATE , 400
EK2_EAS_M_NSE , 1.4
EK2_FLOW_M_NSE , 0.25
EK2_GBIAS_P_NSE , 0.0001
EK2_GSCL_P_NSE , 0.0005
EK2_GYRO_P_NSE , 0.03
EK2_HGT_I_GATE , 500
EK2_MAGB_P_NSE , 0.0001
EK2_MAGE_P_NSE , 0.001
EK2_MAG_I_GATE , 300
EK2_MAG_M_NSE , 0.05
EK2_MAX_FLOW , 2.5
EK2_POS_I_GATE , 500
EK2_RNG_I_GATE , 500
EK2_RNG_M_NSE , 0.5
EK2_VELD_M_NSE , 0.7
EK2_VEL_I_GATE , 500
EK2_WIND_P_NSE , 0.1
EK2_YAW_I_GATE , 300
EK2_YAW_M_NSE , 0.5

The maths that make this system tick are way over my head. But at the minimum I’m going to say whatever it’s trying to do has a serious anomaly.


are you sure that when the mission was created that the mission was set in relative and not absolute heights?
That was my problem.

Sorry, I am editing this now.
I was away and did not follow this from the beginning. It is a small flat area. Forget what I wrote.

(Chris Olson) #12

Yes. All the altitudes in the flight plan are relative to home. This particular flight is the best one to look at for analysis because it is on flat terrain. And it flies the same waypoints over and over again with a DO_JUMP with the altitude gradually bleeding off as it flies the same waypoints repeatedly. There is no barometer drift, no change in barometric pressure, no change in temperature, There is no problem with the hardware because I reverted to AC3.3.3 and flew the same flight without the issue happening. And the issue is repeatable. It happens at high flight speeds in missions with all spline waypoints making turns. The issue has been noted by multi pilots, as well, back to 3.4:

I’m pretty sure it is a flaw in the EKF2 altitude estimator with an error that grows over time flying under these conditions. Since most people don’t fly that way it’s not reported often enough to worry about. And plane has a different implementation than copter has. And the documentation is inaccurate. The altitude in auto obviously does not use the barometer in 3.5.

(pmshop) #13

Found out that Terrain Spline Waypoints work.
Maintained altitude perfectly.
AC 3.5.2

(Chris Olson) #14

Same here. If I fly a mission where there is limited turning there is no issue. For me it only happens if the speed gets high enough to where the heli “cuts off” waypoints (due to speed and acceleration limits) and continuous turning on the mission. I think I have enough data on it to be able to submit a proper issue on GitHub so the EKF experts can take a look at it. I’m thinking some kind of error or “leak” in the code that causes a gradual buildup of the error with a resulting decay of the altitude estimate. I think it’s showing the desired altitude at the final waypoint as being way low because the error gradually “leaked” to where it thought the sensors drifted that far, resulting in the altitude estimate also drifting.

I eliminated hardware issues by reverting to 3.3.3 and flying the same mission with no issues. And it is repeatable on the mission illustrated above, although the drift in altitude is not always the same amount.

When I looked at your issue before @pmshop I was convinced it had to be something else. Because the logs don’t really show what’s going on or indicate any error. Everything looks rosy - right up to the point where the aircraft flies into the ground. The ground station indicates proper altitude, and everything, during the flight. If I went back and looked over your logs now, with what I have learned by analyzing mine, I’m betting I’d see the same anomaly. It was much harder to pick out on your flights because it was over changing terrain elevation. My documented “problem flight” is on perfectly flat terrain where the decay in the actual altitude can be easier seen thru the baro readings, visual reference, and onboard camera reference…

(priseborough) #15

@ChrisOlson Please provide a log and I will have a look at it. From the description and the fact the Baro alt tracks the GPS alt in flight, it could be that the function that slowly adjusts the EKF origin to maintain consistency with GPS is responsible. This behaviour was modified by PR which made it into 3.5.3.

This PR enabled the origin height adjustment to be disabled via the EK2_OGN_HGT_MASK parameter. It should be disabled if EK2_OGN_HGT_MASK = 0.

(priseborough) #16

@ChrisOlson given that your parameter list doesn’t contain the EK2_OGN_HGT_MASK parameter, you must be using 3.5.2 or earlier. If so, then updating to 3.5.3 is recommended.

(Chris Olson) #17

Thanks for taking a look. I wanted to make sure it’s not a simple settings problem.

I’m definitely flying 3.5.3. But you are correct in that there is no EK2_OGN_HGT_MASK param.

This is the BIN log from the described flight:

This is the tlog from QGroundControl:

This is the flight plan from QGroundControl:

Note the flight plan does not use a speed command. WPNAV_SPEED is adjusted on-the-fly with Channel 6. The flight plan is also in QGroundControl’s new JSON file format. If you would like that exported in the more common .txt format I probably have it in another computer with APM Planner2.

The current params are in the BIN log. But I got those in another computer too. I’ll upload them as soon as I get it out of the other PC.

(Chris Olson) #18

@priseborough I just connected with the flight controller to export the params. QGroundControl identifies the firmware version as 3.5.3

But unfortunately I do not see that EK2_OGN_HGT_MASK param in the list

This is the params exported from the Flight Controller…
Scratch that one. Tower has an issue exporting params sometimes. I’ll export from QGroundControl and upload it.

QGroundControl has some extra fields in the params file, but it still should be readable by any text editor. And interestingly it identifies the vehicle type as a multi-rotor instead of a heli frame in the comments header:

(Chris Olson) #19

Paul, I read thru the review on that PR. And it was merged into master (3.6-dev). But did it actually get into 3.5 backports? I checked out the 3.5 branch and I don’t see in there for 3.5.3.

(priseborough) #20

OK, my mistake. I saw the Copter 3.5.3 release commit in master and assumed that was the same as AC3.5.3 release. It isn’t. The AC3.5.3 release is in a separate branch and does not contain

The log data is consistent with an EKF origin that is slowly changing through flight to be consistent with the GPS. The EKF height is tracking the baro correctly and the local vertical position is tracking the control system demand. Both are trending down in height, but the average GPS height remains constant through the mission.

@rmackay9 We need to get that PR into AC3.5.4 ASAP