Looks like AP3.5 GUIDED mode crashed my plane (solved by AP 3.5.2)

I guess it was inevitable, I guess I should have stopped trusting GUIDED mode altogether as long as github.com/diydrones/ardupilot/issues/3582 too.
Still, I wanted to try AP 3.5 and EKF2 to see if somehow they would have fixed my problem. I guess that test is going to cost me a new plane, and many days of building…

I was coming in for landing and forgot to disable geofencing, so when I hit 70 meters altitude, guided mode came on. It’s not supposed to be a big deal, but again it utterly failed and just lost a lot of altitude

Sadly, my OSD camera recording failed, and looks like droneshare is going to be busted forever, so I wasn’t able to upload my log and retrace the flight there.
Baring all that,

I come in for landing, cut off the power, turn to bleed off altitude with crow flaps enabled, come down for landing not realizing that guided took over and is diving towards the ground with the motor on. By the time I realize I’m too high and that I’m not the one flying, I didn’t get to the switch to turn off guided quickly enough and it crashed into a tree, shattering my plane, and weeks of building work and tuning :frowning:

Please tell me if I’m wrong:
Basic log analysis shows that at least GUIDED put full motor power apparently, good.
But it thought the plane was too slow (it did get below 10m/s with a minimum speed of 12m/s), however when speed is fast enough again, and the plane continues to descend, from the logs I’m looking at, airspeed was more than 25m/s when the plane was finally level (not climbing yet) and hit the tree.

If I got this right, this is very disappointing, GUIDED caused the exact crash it was supposed to avoid :frowning:
If not, please let me know what I missed.


Log File N:/apmplanner2/dataflashLogs/bfg18_baylands_crash/21.log
Size (kb) 66153.0537109375
No of lines 555889
Duration 0:07:52
Vehicletype ArduPlane
Firmware Version V3.5.0
Firmware Hash 379ca5ac
Hardware Type
Free Mem 0
Skipped Lines 0
Log analysis shows
Test: Autotune = NA -
Test: Balance/Twist = NA -
Test: Brownout = GOOD -
Test: Compass = FAIL - Large change in mag_field (83.06%)
Test: Dupe Log Data = GOOD -
Test: Empty = GOOD -
Test: Event/Failsafe = GOOD -
Test: GPS = GOOD -
Test: IMU Mismatch = FAIL - Check vibration or accelerometer calibration. (Mismatch: 1.50, WARN: 0.75, FAIL: 1.50)
Test: Parameters = GOOD -
Test: PM = NA -
Test: Pitch/Roll = NA -
Test: Thrust = NA -
Test: VCC = GOOD -

This shows that the altitude had decayed to 22 meters (from 70 when guided was supposed to kick in), and that the plane was apparently on its way to a crash towards the ground if it hadn’t hit that tree.

Why on earth was it going for an airspeed that fast?
Are other flight modes similarly buggy, or just GUIDED?

It didn’t want an airspeed that fast. Graph the demanded airspeed it wanted 12 then it started to want to pitch back up (and just generally had a really poor response to the attempts to pitch up). Copied my comments from github below since I now registered just to respond to this.

[quote]Looking at that my
first thought is that pitching down was absolutely correct, and you’re
pitch loop looks like it never stabilizes, and is not good at handling a
large upset in demand. The airspeed gets as low as 7.5 m/s which is far
outside of the expected/tolerated range there so it was asking for 12 m/s.
As soon as it got that it started looking to kill off the descent and for a
climb (admittedly not as fast as I’d have expected it to).

Also worth noting you had more throttle available in FBWA then the
autopilot ever had during the crash sequence. (I’m not terribly experienced
with STICK_MIXING but the neutral input with FBWA mixing seems like that
wouldn’t be helping with the demands).

I’m not the best log analysis source so I’ll bow to others on that aspect,
but once it got the desired airspeed it started trying to fix it as far as
I can see.[/quote]

damn, sorry for the crash.
I’ve grabbed the log and started to take a look. It does look similar to the last log you gave me. I spent several hours looking at that and didn’t find a clear cause.
I’ll spend some more time on this log and let you know. This is damn frustrating

I think it has got to be something related to the target airspeed (TRIM_ARSPD_CM) being the same as the minimum airspeed (ARSPD_FBW_MIN). I’m still looking …

thanks for chiming in Michael
I’m focussing at the moment on this line of code:
github.com/diydrones/ardupilot/ … S.cpp#L532
I think the _underspeed condition may have latched to true during the initial part of the guided, then stayed there due to the above logic, and height stays below demanded height. That would explain why throttle demand stays at max even after speed is reached, but it doesn’t explain why the pitch demand stays low.
I agree that the aircraft doesn’t respond ideally to the demanded attitude, but I don’t think that is the whole story.

Marc: Guided wasn’t using full throttle because you told the autopilot it wasn’t allowed to use more then 80% throttle, which will be respected in every mode (except takeoff where you can set a different max, or where you control the throttle directly)

Tridge: That might be the case, I’ll have to look through all the under speed behavior once I’m done this test flight.

I’m not sure if you are interested in this Marc, and it’s not the fix to you’re crash but I fly with the override set so that when I am in manual the autopilot will remain in manual regardless of any events (only when I lose rc signal or am in a mode other then manual can the autopilot change modes on me). That would have prevented the inadvertent change to guided, but I’m not sure if that’s behavior you really want?

Marc, can you tell us what channel/channels your flaps were on?
I’m trying to work out why at full throttle and descending with close to neutral pitch the plane didn’t gain speed initially. It did gain speed after a while, but didn’t manage to pitch up even though it wanted to. Could the flaps have a down-pitch effect?

hi Marc,
I’ve been through this log in detail now, and also talked to Paul about controller changes that we can make to avoid it.
First off, here is the key graph that shows all of the features that led to the crash:
It is a very busy graph, so I’ll try and explain the key features.
This graph covers only the time your plane was in GUIDED on the fence breach. The red line shows the height, which initially rises a bit, then descends from 70m down to zero.
The orange line shows TECS.sp, which is the TECS version of the airspeed estimate. This is a smoothed version of the airspeed, converted to a true airspeed. You can see airspeed gets as low as 7m/s at one point.
The light brown line is TECS.sp_dem, which is the TECS demanded airspeed. That doesn’t change much from the configured minimum airspeed.
The black line is PIDP.I, which is the pitch integrator. That plays a vital role in the crash. More on that later.
The grey line is RCOU.Ch2 which is the elevator servo output. The trim elevator was 1532, and numbers below 1532 are demanding up elevator.
The yellow line is throttle, where 1800 is maximum throttle (due to THR_MAX being 80).

The sequence of events is as follows:

  1. fence breach, and enter guided mode, trying to climb
  2. the TECS controller immediately demands pitch up and starts ramping the throttle up. We are now at 9:26:12 on the graph X time axis
  3. the plane responds and does initially pitch up. It also starts to lose airspeed.
  4. at 9:26:13 it enters an underspeed condition as the airspeed drops below the critical ARSPD_FBW_MIN level. Throttle is immediately pegged at maximum (which is 80%) and pitch demand goes down to try to gain speed.
  5. At this point we see the first breakdown of pitch control. The demanded pitch (blue) and actual pitch (green) start to separate. The pitch controller is not managing to produce the down pitch that is wanted. The pitch controller starts to pile on more down elevator (grey line) to try to produce the down pitch that is wanted. Eventually at 9:26:15 the pitch does come briefly under control.

Now I should explain a bit about the underspeed state which the plane is in at this point. Underspeed is declared when airspeed is below 0.9 of the minimum airspeed. Once underspeed is declared then throttle goes to maximum and the TECS pitch controller tries to control airspeed, aiming for the minimum configured airspeed (which is 12m/s for this plane). The idea is that a plane at maximum throttle with airspeed at minimum airspeed should produce a maximum safe climb without stalling. The underspeed condition is not cleared until the plane reaches its desired altitude.

This emergency condition assumes that the plane will respond properly to pitch control while at maximum throttle, and this is where things really start to go wrong in this flight. The TECS controller is demanding down pitch at full throttle but the plane doesn’t initially start to speed up. It is not till several seconds later (at 9:26:21) that the plane finally start to gain speed beyond the target speed.

When it does finally gain speed the TECS controller starts to release the downpitch demand, and starts demanding positive pitch at 9:26:23. The elevator is initially still down, and that is because the pitch controller integrator (the black line) has wound up. The TECS controller ends up saturating at a demanded positive pitch angle of 15 degrees (at 9:26:24) but by this time the pitch controller has completely lost control of the pitch. The actual pitch (green) bears very little resemblance to the demanded pitch (blue).

There are two causes of this. One is the pitch integrator. The PTCH2SRC_I value is quite low, which implies a long slow time constant. It takes a long time for that integrator to unwind and slowly apply more up elevator. The second problem is the pitch gain (the P value) is too low. That can be seen earlier in the flight when pitch control is also not good.

ok, so what can we do about it? I think we will be making several code changes because of this, although not perhaps the ones you may be expecting.

The first thing Paul and I want to add is a detector for “full throttle is pushing the nose down”. It looks like when the underspeed condition triggered full throttle that it caused pitch control to get much worse. The plane just can’t handle sustained full throttle while maintaining pitch control. What we are thinking of doing is adding a rule like this:

  • if the following conditions are all true for 2 seconds:
    • we are in underspeed condition
    • we are more than 20% above the minimum speed
    • we are descending

then we will cut throttle back to the TRIM_THROTTLE, only allowing it to slew back up over another 2 seconds. This should give the airframe time to recover from the “too high throttle” state.

The second thing we are going to do is work out a minimum acceptable level of I term for the pitch controller (ie. PTCH2SRV_I). Too many issues are caused by the very slow response of the integrator in pitch, as happens in this flight. We will set the minimum I term to correspond to a time constant of around 3 seconds. That should reduce the problems of TECS not getting the pitch it is demanding.

The log doesn’t actually show a TECS bug. TECS is actually operating as it was designed to operate. It just assumed that airframes respond in a reasonable manner, and some airframes don’t respond well to full throttle.

I would still be interested in knowing what your flaps setup was on this plane, just because there is a possibility that flaps contributed to the loss of pitch control.


First, Tridge and WickedShell, thanks for your time looking at this, I do appreciate it.
Sorry for my delayed reply, I forgot that this forum does not send Email to warn you you got an answer to your thread (by default). Kind of a misfeature…

Random replies:

  • I had TRIM_ARSPD_CM at 15 and ARSPD_FBW_MIN at 12 last time we talked Tridge, and I seem to remember that you recommended I lower TRIM_ARSPD_CM to the same value than ARSPD_FBW_MIN, so I did before this flight and crash.

  • You did mention I got as low as 7.5m/s, as per my reply in the bug, that’s not what I found in the logs, nor do I believe it’s possible. This plane stalls below 11m/s. But as agreed, the airplane did get too slow and did at least for a while need pitch down to maintain flying airspeed. The AP made sure it never stalled.

  • You’re correct about full throttle being only 80% per my request in the config, I had half forgotten about that, but at the same time on that motor 80% was mostly maximum thrust anyway, and I didn’t want the motor to be overstressed and burnt during normal flight. You could make the case that allowing takeoff power might be allowable for GUIDED with a minimum altitude deck, would be a good thing, but that’s half a detail.

  • Sorry, I forgot to give my channel mapping
    o CH01 : Right Aileron
    o CH02 : Elevator
    o CH03 : Throttle
    o CH04 : Rudder
    o CH05 : left aileron
    o CH06 : R flap
    o CH07 : L flap
    o CH08 : OSDCmg: up 12V+cams off, mid: 12V on, down: GUIDED. 2 mode flips: OSD rotate
    o CH09 : FPV Camera L/R (S2)
    o CH10 : free
    o CH11: RSSIQual from RX
    o CH12 : Ardupilot Mode Switch (6 position switch mod)
    I’m pretty sure that both flaps were full down due to manual passthrough

  • Having override set in manual, yes, it’s a reasonable idea. I kind of wanted GUIDED to save me in case I did something stupid, and lost control, but right now it’s more likely to crash my plane, than me, so manual removing the geofence is not a bad idea

  • Putting crow flaps actually causes my plane ot pitch up, I have a crow flaps to pitch down mix. However I never fly with flaps only (my controller is not setup for it), so I cannot swear to you that flaps cannot cause pitch down, but I’m pretty certain they won’t because every single full size plane I’ve flown requires trim down as you add flaps, in other words flaps cause nose up.
    However full flaps down on that plane will definitely add a fair amount of drag (that’s the point), so trying to fly back up with full flaps should be possible, but harder. Given that the airspeed went as fast as 26m/s with flaps down, I assume some of that energy could easily have been used to fly at least level, or back up.

I read your detailled analysis Tridge (thanks), and you also found that keeping pitch down was harder with the flaps, which agrees with flap causes pitch up. On that plane full throttle also causes pitch up, but not horribly so. When I fly in FBWA or CRUISE, the AP seems to handle that condition transparently and keeps things in check ok.
So, I do think the flaps caused loss of pitch control much more than throttle because I used full allowed throttle in FBWA/CRUISE many times without problems

Tridge, to see if throttle was more an issue than flaps, can you compare with
marc.merlins.org/tmp/6_GUIDED_st … _crash.BIN
where I went from FBWA to GUIDED and a similar looking issue happened except no flaps should have been involved then.
This will help validate (on the same airframe) if your analysis and fixes would have helped with the issue there too

Actually another data point that could be useful for analysis:
AP 3.4: RTL and CRUISE both give occasional oscillations and stalls of death

youtube.com/watch?v=UnuNdcZ … be&t=18m5s

Looking at it again, I don’t think it’s a pitch up due to throttle because not much throttle was applied, and I don’t think RTL motor power up would have caused what was a perfectly flying plane above RTL target altitude to start flying so badly, but since we’re talking about control loop issues, this may still be worth looking into, to see if it’s related.

Airframe is a BFG 2600 which a T tail aircraft. Elevator has always had plenty of authority in all flight conditions as far as I can tell

Hi Tridge,
My apologies if this is totally unrelated, but I’m bringing it up in case it can help validate your recent fix.
On my last post about github.com/diydrones/ardupilot/issues/3032 you may want to scroll to
’marcmerlin commented on Oct 18, 2015
Actually I was just able to retrieve some screenshots from the OSD (recording is incomplete, but those are enough to help).’

Lower down in that bug:
Example 1: CRUISE failure:
I’m not sure why CRUISE was commanding such a pitch up attitude causing repeated stalls, and with only 22% throttle. Usually it puts a lot of power when it goes up, and here virtually none (I see 1.93A draw, that means the motor is barely turning).
So why

  1. so much pitch up
  2. no motor power
  3. no detection of the stalls and attempt to regain

Example 2: RTL failure
Pausing on a non snowy frame shows pitch up of 23 degrees, no idea why, and RTL missing the target heading by more than 90 degrees.
I can see how eventually the airspeed drops enough to do this, but there is no reason for RTL to fly so erratically with such a nose up altitude and not full power. And if it gets into that mode, shouldn’t it detect the stalls, and add power?
Switch to CRUISE at 2:19 fixes the problem

There was the issue that ARSPD_FBW_MIN had been raised to 14 to see if it helped flight behaviour while TRIM_ARSPD_CM was still 12, but there still seems to be a pitch control issue here.

For anyone reading this thread, AP 3.5.2 has a fix that is supposed to prevent the crash that happened here.
Given spare time (always in short supply obviously), @tridge is going to have a look at the other logs with weird pitch issues to see if they were related and most likely addressed by the fix he committed.
Thanks again @tridge for the code fix and reviewing the logs.

@tridge , I just did some test flights with 3.5.2 today, with full flaps down, breached the low altitude fence, and let GUIDED take over a few times.
While it struggled to stop the descent the bit and regain altitude (and that’s expected, the flaps are big), it did the right thing and went back up.
So at this point, I have some confidence that your fix did indeed fix something :slight_smile:
The other logs I gave in this bug may still point to some separate issues since the last near crash happened without the flaps down if I recall correctly, but at least the GUIDED takes over with flaps down case seems to work ok now.

Also, I have crow fflaps with differential throws and going to 130%, so I’m not sure how possible or easy it is to set them up as flaps controlled by AP, but if I do, should I assume that GUIDED wiill do the right thing and retract the flaps when it needs to fly back up?

@tridge, you asked for logs showing GUIDED fly back up with the flaps down.
I just did this today,

At 3:22, I disabled geofence, used MANUAL and put full flaps to drop below 100m,
guided takes over and climbs back to 100m

So that’s good :slight_smile:
What’s not so good was that my fence return altitude was 150m, and it only went back to 100m
PARM, 651332702, FENCE_ACTION, 1
PARM, 651332718, FENCE_TOTAL, 6
PARM, 651332734, FENCE_CHANNEL, 8
PARM, 651332839, FENCE_MINALT, 100
PARM, 651332855, FENCE_MAXALT, 1000
PARM, 651332870, FENCE_RETALT, 150
PARM, 651332902, FENCE_RET_RALLY, 1

I did not have a rally point, so it went back to the takeoff point, which is good, but ignored the 150m and went back to 100m.
Any idea why?
Then, there are clear issues with airspeed, but I put that in another message: Why does airspeed not reset to 0 before takeoff and why did autocal make it worse?

Mmmh, actually in my next flight where I crashed RTL failed to recover from inverted flight and spun backwards into the ground check
https://youtu.be/wFwTneYeNY4?t=06:52s (offset 06:52)
CRUZ is hoping up and down, which looks a biti weird

@tridge you can load the 57s CRUIISE segment between GUIDED 1s and FBWA, it’s the part where the plane keeps nudging up and down.
I had a look with Mavproxy, but I’m not sure if I’m figuring out what’s wrong. Is it a problem with PID? I had flown that airframe a fair bit since autotune, and never experienced this up/down problem (actually I think it didn’t happen for the rest of the flight)

Looking at these graphs, all I seem to see is AP commanding that pitch to go up and down

@tridge more reading seems to say that PTCH2SRV_P could cause porpoise.
Looks like autotune gave me 0.92 for it, not sure if it’s too high.
Still, weird that I never had porpoise behaviour until this flight.

PTCH2SRV_D 0.074393
PTCH2SRV_FF 0.000000
PTCH2SRV_I 0.062053
PTCH2SRV_IMAX 2000.000000
PTCH2SRV_P 0.929917
PTCH2SRV_RLL 1.000000
PTCH2SRV_RMAX_DN 90.000000
PTCH2SRV_RMAX_UP 90.000000

I guess when I get my plane repaired after its last crash, I can try the steps from http://ardupilot.org/plane/docs/roll-pitch-controller-tuning.html if autotune isn’t doing quite the right thing for my plane anymore.

Is RTL supposed to auto-land? Or does the plane circle overhead? Thanks.

@ripfree: you are totally asking this in the wrong place. It circles overhead. Please use a new thread for questions, don’t hijack another one :slight_smile: