thanks for chiming in Michael
I’m focussing at the moment on this line of code:
github.com/diydrones/ardupilot/ … S.cpp#L532
I think the _underspeed condition may have latched to true during the initial part of the guided, then stayed there due to the above logic, and height stays below demanded height. That would explain why throttle demand stays at max even after speed is reached, but it doesn’t explain why the pitch demand stays low.
I agree that the aircraft doesn’t respond ideally to the demanded attitude, but I don’t think that is the whole story.
Marc: Guided wasn’t using full throttle because you told the autopilot it wasn’t allowed to use more then 80% throttle, which will be respected in every mode (except takeoff where you can set a different max, or where you control the throttle directly)
Tridge: That might be the case, I’ll have to look through all the under speed behavior once I’m done this test flight.
I’m not sure if you are interested in this Marc, and it’s not the fix to you’re crash but I fly with the override set so that when I am in manual the autopilot will remain in manual regardless of any events (only when I lose rc signal or am in a mode other then manual can the autopilot change modes on me). That would have prevented the inadvertent change to guided, but I’m not sure if that’s behavior you really want?
Marc, can you tell us what channel/channels your flaps were on?
I’m trying to work out why at full throttle and descending with close to neutral pitch the plane didn’t gain speed initially. It did gain speed after a while, but didn’t manage to pitch up even though it wanted to. Could the flaps have a down-pitch effect?
hi Marc,
I’ve been through this log in detail now, and also talked to Paul about controller changes that we can make to avoid it.
First off, here is the key graph that shows all of the features that led to the crash:
http://uav.tridgell.net/MarcMerlin/guided_crash.png
It is a very busy graph, so I’ll try and explain the key features.
This graph covers only the time your plane was in GUIDED on the fence breach. The red line shows the height, which initially rises a bit, then descends from 70m down to zero.
The orange line shows TECS.sp, which is the TECS version of the airspeed estimate. This is a smoothed version of the airspeed, converted to a true airspeed. You can see airspeed gets as low as 7m/s at one point.
The light brown line is TECS.sp_dem, which is the TECS demanded airspeed. That doesn’t change much from the configured minimum airspeed.
The black line is PIDP.I, which is the pitch integrator. That plays a vital role in the crash. More on that later.
The grey line is RCOU.Ch2 which is the elevator servo output. The trim elevator was 1532, and numbers below 1532 are demanding up elevator.
The yellow line is throttle, where 1800 is maximum throttle (due to THR_MAX being 80).
The sequence of events is as follows:
- fence breach, and enter guided mode, trying to climb
- the TECS controller immediately demands pitch up and starts ramping the throttle up. We are now at 9:26:12 on the graph X time axis
- the plane responds and does initially pitch up. It also starts to lose airspeed.
- at 9:26:13 it enters an underspeed condition as the airspeed drops below the critical ARSPD_FBW_MIN level. Throttle is immediately pegged at maximum (which is 80%) and pitch demand goes down to try to gain speed.
- At this point we see the first breakdown of pitch control. The demanded pitch (blue) and actual pitch (green) start to separate. The pitch controller is not managing to produce the down pitch that is wanted. The pitch controller starts to pile on more down elevator (grey line) to try to produce the down pitch that is wanted. Eventually at 9:26:15 the pitch does come briefly under control.
Now I should explain a bit about the underspeed state which the plane is in at this point. Underspeed is declared when airspeed is below 0.9 of the minimum airspeed. Once underspeed is declared then throttle goes to maximum and the TECS pitch controller tries to control airspeed, aiming for the minimum configured airspeed (which is 12m/s for this plane). The idea is that a plane at maximum throttle with airspeed at minimum airspeed should produce a maximum safe climb without stalling. The underspeed condition is not cleared until the plane reaches its desired altitude.
This emergency condition assumes that the plane will respond properly to pitch control while at maximum throttle, and this is where things really start to go wrong in this flight. The TECS controller is demanding down pitch at full throttle but the plane doesn’t initially start to speed up. It is not till several seconds later (at 9:26:21) that the plane finally start to gain speed beyond the target speed.
When it does finally gain speed the TECS controller starts to release the downpitch demand, and starts demanding positive pitch at 9:26:23. The elevator is initially still down, and that is because the pitch controller integrator (the black line) has wound up. The TECS controller ends up saturating at a demanded positive pitch angle of 15 degrees (at 9:26:24) but by this time the pitch controller has completely lost control of the pitch. The actual pitch (green) bears very little resemblance to the demanded pitch (blue).
There are two causes of this. One is the pitch integrator. The PTCH2SRC_I value is quite low, which implies a long slow time constant. It takes a long time for that integrator to unwind and slowly apply more up elevator. The second problem is the pitch gain (the P value) is too low. That can be seen earlier in the flight when pitch control is also not good.
ok, so what can we do about it? I think we will be making several code changes because of this, although not perhaps the ones you may be expecting.
The first thing Paul and I want to add is a detector for “full throttle is pushing the nose down”. It looks like when the underspeed condition triggered full throttle that it caused pitch control to get much worse. The plane just can’t handle sustained full throttle while maintaining pitch control. What we are thinking of doing is adding a rule like this:
- if the following conditions are all true for 2 seconds:
- we are in underspeed condition
- we are more than 20% above the minimum speed
- we are descending
then we will cut throttle back to the TRIM_THROTTLE, only allowing it to slew back up over another 2 seconds. This should give the airframe time to recover from the “too high throttle” state.
The second thing we are going to do is work out a minimum acceptable level of I term for the pitch controller (ie. PTCH2SRV_I). Too many issues are caused by the very slow response of the integrator in pitch, as happens in this flight. We will set the minimum I term to correspond to a time constant of around 3 seconds. That should reduce the problems of TECS not getting the pitch it is demanding.
The log doesn’t actually show a TECS bug. TECS is actually operating as it was designed to operate. It just assumed that airframes respond in a reasonable manner, and some airframes don’t respond well to full throttle.
I would still be interested in knowing what your flaps setup was on this plane, just because there is a possibility that flaps contributed to the loss of pitch control.
3 Likes
First, Tridge and WickedShell, thanks for your time looking at this, I do appreciate it.
Sorry for my delayed reply, I forgot that this forum does not send Email to warn you you got an answer to your thread (by default). Kind of a misfeature…
Random replies:
-
I had TRIM_ARSPD_CM at 15 and ARSPD_FBW_MIN at 12 last time we talked Tridge, and I seem to remember that you recommended I lower TRIM_ARSPD_CM to the same value than ARSPD_FBW_MIN, so I did before this flight and crash.
-
You did mention I got as low as 7.5m/s, as per my reply in the bug, that’s not what I found in the logs, nor do I believe it’s possible. This plane stalls below 11m/s. But as agreed, the airplane did get too slow and did at least for a while need pitch down to maintain flying airspeed. The AP made sure it never stalled.
-
You’re correct about full throttle being only 80% per my request in the config, I had half forgotten about that, but at the same time on that motor 80% was mostly maximum thrust anyway, and I didn’t want the motor to be overstressed and burnt during normal flight. You could make the case that allowing takeoff power might be allowable for GUIDED with a minimum altitude deck, would be a good thing, but that’s half a detail.
-
Sorry, I forgot to give my channel mapping
o CH01 : Right Aileron
o CH02 : Elevator
o CH03 : Throttle
o CH04 : Rudder
o CH05 : left aileron
o CH06 : R flap
o CH07 : L flap
o CH08 : OSDCmg: up 12V+cams off, mid: 12V on, down: GUIDED. 2 mode flips: OSD rotate
o CH09 : FPV Camera L/R (S2)
o CH10 : free
o CH11: RSSIQual from RX
o CH12 : Ardupilot Mode Switch (6 position switch mod)
I’m pretty sure that both flaps were full down due to manual passthrough -
Having override set in manual, yes, it’s a reasonable idea. I kind of wanted GUIDED to save me in case I did something stupid, and lost control, but right now it’s more likely to crash my plane, than me, so manual removing the geofence is not a bad idea
-
Putting crow flaps actually causes my plane ot pitch up, I have a crow flaps to pitch down mix. However I never fly with flaps only (my controller is not setup for it), so I cannot swear to you that flaps cannot cause pitch down, but I’m pretty certain they won’t because every single full size plane I’ve flown requires trim down as you add flaps, in other words flaps cause nose up.
However full flaps down on that plane will definitely add a fair amount of drag (that’s the point), so trying to fly back up with full flaps should be possible, but harder. Given that the airspeed went as fast as 26m/s with flaps down, I assume some of that energy could easily have been used to fly at least level, or back up.
I read your detailled analysis Tridge (thanks), and you also found that keeping pitch down was harder with the flaps, which agrees with flap causes pitch up. On that plane full throttle also causes pitch up, but not horribly so. When I fly in FBWA or CRUISE, the AP seems to handle that condition transparently and keeps things in check ok.
So, I do think the flaps caused loss of pitch control much more than throttle because I used full allowed throttle in FBWA/CRUISE many times without problems
Tridge, to see if throttle was more an issue than flaps, can you compare with
marc.merlins.org/tmp/6_GUIDED_st … _crash.BIN
from
github.com/diydrones/ardupilot/issues/3357
where I went from FBWA to GUIDED and a similar looking issue happened except no flaps should have been involved then.
This will help validate (on the same airframe) if your analysis and fixes would have helped with the issue there too
Actually another data point that could be useful for analysis:
github.com/diydrones/ardupilot/issues/3032
AP 3.4: RTL and CRUISE both give occasional oscillations and stalls of death
marc.merlins.org/tmp/MANU_to_GUIDED_4.BIN
youtube.com/watch?v=UnuNdcZ … be&t=18m5s
Looking at it again, I don’t think it’s a pitch up due to throttle because not much throttle was applied, and I don’t think RTL motor power up would have caused what was a perfectly flying plane above RTL target altitude to start flying so badly, but since we’re talking about control loop issues, this may still be worth looking into, to see if it’s related.
Airframe is a BFG 2600 which a T tail aircraft. Elevator has always had plenty of authority in all flight conditions as far as I can tell
Hi Tridge,
My apologies if this is totally unrelated, but I’m bringing it up in case it can help validate your recent fix.
On my last post about github.com/diydrones/ardupilot/issues/3032 you may want to scroll to
’marcmerlin commented on Oct 18, 2015
Actually I was just able to retrieve some screenshots from the OSD (recording is incomplete, but those are enough to help).’
Lower down in that bug:
Example 1: CRUISE failure:
youtu.be/y-TKmxJ5Tyg?t=10s
I’m not sure why CRUISE was commanding such a pitch up attitude causing repeated stalls, and with only 22% throttle. Usually it puts a lot of power when it goes up, and here virtually none (I see 1.93A draw, that means the motor is barely turning).
So why
- so much pitch up
- no motor power
- no detection of the stalls and attempt to regain
marc.merlins.org/tmp/Cruise_Stall_4.BIN
Example 2: RTL failure
youtu.be/uD5Ou2tTKjM?t=1m41s
Pausing on a non snowy frame shows pitch up of 23 degrees, no idea why, and RTL missing the target heading by more than 90 degrees.
I can see how eventually the airspeed drops enough to do this, but there is no reason for RTL to fly so erratically with such a nose up altitude and not full power. And if it gets into that mode, shouldn’t it detect the stalls, and add power?
Switch to CRUISE at 2:19 fixes the problem
marc.merlins.org/tmp/RTL_Stall_3.BIN
There was the issue that ARSPD_FBW_MIN had been raised to 14 to see if it helped flight behaviour while TRIM_ARSPD_CM was still 12, but there still seems to be a pitch control issue here.
For anyone reading this thread, AP 3.5.2 has a fix that is supposed to prevent the crash that happened here.
Given spare time (always in short supply obviously), @tridge is going to have a look at the other logs with weird pitch issues to see if they were related and most likely addressed by the fix he committed.
Thanks again @tridge for the code fix and reviewing the logs.
@tridge , I just did some test flights with 3.5.2 today, with full flaps down, breached the low altitude fence, and let GUIDED take over a few times.
While it struggled to stop the descent the bit and regain altitude (and that’s expected, the flaps are big), it did the right thing and went back up.
So at this point, I have some confidence that your fix did indeed fix something 
The other logs I gave in this bug may still point to some separate issues since the last near crash happened without the flaps down if I recall correctly, but at least the GUIDED takes over with flaps down case seems to work ok now.
Also, I have crow fflaps with differential throws and going to 130%, so I’m not sure how possible or easy it is to set them up as flaps controlled by AP, but if I do, should I assume that GUIDED wiill do the right thing and retract the flaps when it needs to fly back up?
@tridge, you asked for logs showing GUIDED fly back up with the flaps down.
I just did this today,
http://marc.merlins.org/tmp/12_GUIDED_FLAPS_CLIMB.BIN
At 3:22, I disabled geofence, used MANUAL and put full flaps to drop below 100m,
https://youtu.be/NgUKE-YJfHg?t=3m45s
guided takes over and climbs back to 100m
So that’s good 
What’s not so good was that my fence return altitude was 150m, and it only went back to 100m
PARM, 651332702, FENCE_ACTION, 1
PARM, 651332718, FENCE_TOTAL, 6
PARM, 651332734, FENCE_CHANNEL, 8
PARM, 651332839, FENCE_MINALT, 100
PARM, 651332855, FENCE_MAXALT, 1000
PARM, 651332870, FENCE_RETALT, 150
PARM, 651332886, FENCE_AUTOENABLE, 0
PARM, 651332902, FENCE_RET_RALLY, 1
I did not have a rally point, so it went back to the takeoff point, which is good, but ignored the 150m and went back to 100m.
Any idea why?
Then, there are clear issues with airspeed, but I put that in another message: Why does airspeed not reset to 0 before takeoff and why did autocal make it worse?
Mmmh, actually in my next flight where I crashed RTL failed to recover from inverted flight and spun backwards into the ground check
https://youtu.be/wFwTneYeNY4?t=06:52s (offset 06:52)
CRUZ is hoping up and down, which looks a biti weird
@tridge you can load the 57s CRUIISE segment between GUIDED 1s and FBWA, it’s the part where the plane keeps nudging up and down.
I had a look with Mavproxy, but I’m not sure if I’m figuring out what’s wrong. Is it a problem with PID? I had flown that airframe a fair bit since autotune, and never experienced this up/down problem (actually I think it didn’t happen for the rest of the flight)
http://marc.merlins.org/tmp/12_GUIDED_FLAPS_CLIMB.BIN
Looking at these graphs, all I seem to see is AP commanding that pitch to go up and down
@tridge more reading seems to say that PTCH2SRV_P could cause porpoise.
Looks like autotune gave me 0.92 for it, not sure if it’s too high.
Still, weird that I never had porpoise behaviour until this flight.
PTCH2SRV_D 0.074393
PTCH2SRV_FF 0.000000
PTCH2SRV_I 0.062053
PTCH2SRV_IMAX 2000.000000
PTCH2SRV_P 0.929917
PTCH2SRV_RLL 1.000000
PTCH2SRV_RMAX_DN 90.000000
PTCH2SRV_RMAX_UP 90.000000
PTCH2SRV_TCONST 0.400000
I guess when I get my plane repaired after its last crash, I can try the steps from http://ardupilot.org/plane/docs/roll-pitch-controller-tuning.html if autotune isn’t doing quite the right thing for my plane anymore.
Is RTL supposed to auto-land? Or does the plane circle overhead? Thanks.
@ripfree: you are totally asking this in the wrong place. It circles overhead. Please use a new thread for questions, don’t hijack another one
absolutely brilliant your analisis above, thanks a lot for contribute to this knowledge.
I have experienced many different behaviors thinking that something is wrong in multicopters and plane but almost always the solution turns out to be a summation of conditions and parameters set correctly . This discipline takes time and patience.
373 01-01-1970 05-30-00.bin (2.0 MB)
Hi, Crashed Skywalker Arduplane 3.40 when switched to RTL while ythe plane was in guided mode “Go to Here”
I can see that there is no Airplane firmware upgrade for APM beyond 3.40 and I can see that the bug was resolved in Arduplane 3.52. Can someone help for a patch for APM Arduplane 3.40 ?
@Rana As you may have seen in more than one announcement, the orginal APM isn’t supported anymore, so unfortunately you shouldn’t really expect firmware updates for your flight controller anymore.
The original APM has been obsolete for more than 2 years now, my recommendation if you care about your airframes is to stick to a pixhawk clone, or better from now on.
Hi,
I have many many APM’s and want to use them, so if either Tridge or someone else can share the patch to take care the issue of crash in guided then I could make use of these APM’s
I have many pixhawk too but want to use the APM also.
@Rana, I had a look at your log (“373 01-01-1970 05-30-00.bin”) and it doesn’t look anything like the issue that Marc previously saw.
The first thing I notice in your log is that no roll or pitch tuning has been done - it is just using the defaults. It looks like the gains are way too low. Because of the very low gains it takes a long time to build up enough elevator to start to pull up when it is descending. The main thing you need to do is run an autotune, or even just manually increase the P and I gains or both roll and pitch. It takes nearly six seconds to build up enough pitch integrator to start to pull up, and that is much too long. You probably need about 5x the integrator gain, and 3x the P gain at a rough guess.
I am happy to do a new bug fix only release for the APM2 if a really serious bug is found and I have a log that shows it is needed, but this is not an example of that sort of problem.
Cheers, Tridge