RTL crash (again)

Log Analysis -- deprecated Copter Logs -- deprecated
1

I’ve had a repeat of an incident a few flights ago where the coper is in RTL and returning nicely when suddenly it descends rapidly and crashes. In both instances I had time to switch to stabilise mode and give it full throttle to attempt a recovery. In both cases there was no response from the throttle input and the copter crashed.

I could use some help figuring out what is causing this. I’ve looked at the flash log/tlog and it looks like the voltage and current were fine. I can’t see any data from the baro readings but I’m suspecting it might be part of the issue.

Any help would be much appreciated. The relevant parts of the log start at about 82% in. Ignore the switch to Autotune at the beginning. I had a switch programmed wrong on my transmitter so I landed, fixed that and took off again.

1 Like
2

Could you get the bin log off of the SD card? The log is corrupt. I am able to fix it to the point where it is graphable, but there is certainly missing data.

3

Here’s the .bin and a .tlog

4

Channels 1, 3, 5, and 7 are 200pwm lower than channels 2, 4, 6, and 8.
I assume you’re flying this configuration:

I don’t think that this should be happening on an X8. Are you sure your motors are connected correctly? I had a recent experience on my Y6 (Y6B config), I had the two motors on one arm swapped and it would fly, but it was yawing around. The arm produced the roll and pitch torques as it was supposed to, but it was producing the opposite yaw torque. An X8 has enough motors to fly like this, but it would be capable of only reduced max throttle.

Is there a way to check for this possibility? Obviously it was flying for a while and then it wasn’t, which means something changed in flight, but there is definitely a problem if the CWs and CCWs are so clearly split on an X8.

5

Ok, so, a few observations here:

  1. You’re hovering at 80% throttle. You don’t have a lot of margin.
  2. You have this yaw offset causing you to lose 10% margin in the second flight in your log, but not the first
  3. You’ve manually tuned the copter and at least your roll D term is too high. You’re getting rapid oscillation (5hz). This rapid oscillation is causing the stability patch to reduce your max throttle even further.

My remaining question is, what configuration changes did you do while the copter was landed? The yaw offset started afterward, but it could be some weird artifact of the oscillation maybe? I’m not certain about that one, I will talk to Leonard.

The conclusions I can reach so far are:

  1. That you need to tune very conservatively. Your copter’s load changes a lot and this is causing you problems. Why don’t you run autotune with no load, drop your STB_ gain to the old defaults (4.5) and see how well it flies with load? As I mentioned before, gimballed load doesn’t really count as it has what amounts to no rotational inertia.
  2. That you should have more power or less weight. You’re losing the redundancy advantage of having an X8. If a motor fails, it will not flip, but it will still crash.
6

Ok, turns out that the yaw offset could not be caused by a motor being swapped (after speaking with Leonard). Its hard to tell what’s causing that without being able to play with the copter myself.

Your fundamental problems are your rate gains and your loading. Fix those, fly it in a hover and send me a log and we’ll see if the yaw thing has gone away and it seems safe. At the moment it is not safe as it will crash if it loses a single motor regardless of gains.

May I ask why you chose X8? An X8 is not a heavy-lift frame, it is a compact medium-lift frame.

7

[quote=“jschall”]Ok, turns out that the yaw offset could not be caused by a motor being swapped (after speaking with Leonard). Its hard to tell what’s causing that without being able to play with the copter myself.

Your fundamental problems are your rate gains and your loading. Fix those, fly it in a hover and send me a log and we’ll see if the yaw thing has gone away and it seems safe. At the moment it is not safe as it will crash if it loses a single motor regardless of gains.

May I ask why you chose X8? An X8 is not a heavy-lift frame, it is a compact medium-lift frame.[/quote]
OK, a few things here to look at then.

My hover rate certainly is not 80% so I’m not sure what indicates that.

What changed after landing? Well, I have an idea. The reason I landed was that I have my camera shutter on Ch 7 which I also had assigned to start the Autotune. I took off, tripped the shutter and it went in to Autotune. So I landed, disabled Autotune on ch 7 and took off again.

But I think you are on to something with the motor order. My props and motor directions are correct. But I just ran the CLI motor test and it went in this order:
1,6,4,7,8,3,2,6

Shouldn’t that be:
1,6,4,7,3,8,2,6 ?

As in 3 and 8 are reversed? Obviously an issue and I’ll swap them on your advice. But certainly no reason for the main two issues I reported:

  1. why does it loose thrust randomly and only when in auto modes? which you have say is probably due to high load. but I don’t think so. it hovers just fine with the stick at 55%.

2). When it does loose thrust why is their no response when I move the stick to full throttle?

I’ve been considering going to 16" props for a bit more thrust which might help.

The reason I run a coax configuration is that amongst video pilots it’s preferred for several reasons. Have a read of MultiRotorForums about it. Those people are all making a living doing very complex shooting for television and movies so I think they make a good reference when it comes to things like this.

  1. There’s a perception that they fly better in the wind
  2. They are physically easier to transport
  3. People who have flown flat 8 and coax 8 have noticed little to no difference in suitability as a video platform.
  4. Droidworx (now Aeronavics) have sold a lot of copters in this configuration and get great feedback from their customers. They are without a doubt one of the most respected manufacturers of high end systems. Their systems go up to the capacity of Red Epic cameras.
  5. A friend of mine has the next coax 8 size down and it’s without a doubt the most stable and best handling copter I’ve ever seen bar none. I’d challenge ANYONE to build a copter that is more stable.
8

[quote]
May I ask why you chose X8? An X8 is not a heavy-lift frame, it is a compact medium-lift frame.[/quote]
I’ll leave that for you to debate with the staff at Droidworx. The copter was built to their specs for the specific purpose of flying a Nikon D600. I’d like to think they know what they are taking about but I do realise that sometimes they can be over optimistic.

I should mention that I fly this copter with a dummy load all the time for practice and to ensure it’s reliable. I know when a copter is too heavy as I’ve flown a few that were overloaded in the past. Honestly, this one appears to handle the load just fine. It has MN 4014-11 motors on it and 15x5 T-Motor carbon fibre props that are quite efficient.

9

[quote=“jschall”]Ok, so, a few observations here:
2. That you should have more power or less weight. You’re losing the redundancy advantage of having an X8. If a motor fails, it will not flip, but it will still crash.[/quote]

About a month ago a top prop flew off while hovering due to being loose (yes, I forgot to check). The copter flew totally normally and there was little to no effect on how it flew. I was just amazed. In stabilise mode I just flew it back to the take off point and landed it normally.

10

Ok. Please check your motor order carefully - the X8 will appear to be flying fine with all sorts of problems.

Fig. 1. You really need more margin than you have - you need 25% or more to even be able to fly with a dead motor.

The stability code prioritizes roll and pitch over everything else. That means that if it has a big roll or pitch demand, it will put throttle at 50% in order to max one side and min the other if it has to. When you get extremely bad oscillations with a hover throttle above 50%, it will do that very frequently and that will cause it to descend.

11

[quote=“dazzab”][quote=“jschall”]Ok, so, a few observations here:
2. That you should have more power or less weight. You’re losing the redundancy advantage of having an X8. If a motor fails, it will not flip, but it will still crash.[/quote]

About a month ago a top prop flew off while hovering due to being loose (yes, I forgot to check). The copter flew totally normally and there was little to no effect on how it flew. I was just amazed. In stabilise mode I just flew it back to the take off point and landed it normally.[/quote]
Sorry, my statement is misleading. I remembered that when this happened it didn’t have any pay load on it. So it would have had plenty of power to spare.

12

Lots of good info here. Thank you for all your help.

I will:

  • Correct the motor order of 3/8
  • Reset to defaults
  • Do a flight to auto tune it (without payload)
  • Check motor alignments - in my experience yaw issues have always been related to this.
  • Double check the hover rate.

I don’t understand why that graph shows it as so high. I would recognise a throttle rate that high and the copter doesn’t exhibit flight characteristics of being that over loaded. I can improve the throttle rate with larger props. I’ve asked both the copter manufacturer and a vendor specialising in video copters their opinion on this. I should mention that eCalc shows that my configuration will hover at 60% even if I input larger weight etc than I’m actually using so this still seems odd.

I am very concerned about how suitable Arducoper is for heavy camera copters after working through this with you. A system really shouldn’t lower a throttle rate to a fixed percentage if that will cause a descent that will damage the equipment. I see the logic of sacrificing thrust for stability but in my case there was no need for stability input given it was flying smoothly so why would it do that in the first place?

If a pilot calls for more thrust, the system shouldn’t ignore that. That’s what the pilot is for, to over ride algorithms that are about to create problems. I should also mention that even though the gains may have been high, the copter was in no way in a flight condition that would cause excessive need for any stabilisation. If the flight controller decided to sacrifice thrust in this situation then it doest seem to me that it’s suitable for use with heavy camera systems at this point in time.

13

It does not lower the throttle to a fixed value, it lowers it in order to meet the roll and pitch demand.

The pilot is going to react to a descent by applying full throttle approximately 100% of the time. Do you want that to cause the copter to lose control of every axis and flip over? I welcome you to read the code - it is in libraries/AC_Motors/MotorsMatrix.cpp, in the output_armed function.

14

[quote=“jschall”]
The pilot is going to react to a descent by applying full throttle approximately 100% of the time. Do you want that to cause the copter to lose control of every axis and flip over? I welcome you to read the code - it is in libraries/AC_Motors/MotorsMatrix.cpp, in the output_armed function.[/quote]

Are you saying that it’s not possible to maintain thrust and control with the copter as it’s configured? The only way to maintain stability is to let it crash land? If that’s the case then Arducopter certainly is not suitable for my purposes.

I’ll bet you a hundred dollars that if I put my Wookong controller on this copter that it won’t crash when it does a RTL. Many people have the exact copter built to the same specs and their copters don’t crash land when doing RTL. So why is it unreasonable for me to expect the same with Arducopter?

I thought you would see this as an opportunity to improve the platform and appreciate my assistance. It’s cost me a lot of time and significant amounts of money to prove that this project is the way of the future for those doing doing professional aerial photography with high end systems.

15

I said ‘fixed percentage’ not ‘fixed value’. You seem to be missing quite a bit of what I’m saying. Such as why would it need to put so much power in to stabilisation in the first place? All it was doing was an RTL on an extremely calm windless day.

[quote=“jschall”]
I welcome you to read the code - it is in libraries/AC_Motors/MotorsMatrix.cpp, in the output_armed function.[/quote]
Providing he is willing and has the time, I’ll ask Tridge to explain it to me. He’s come up with a lot of great ideas for this project so maybe he will see something that can help out in this type of scenario as well. There has to be a better solution than just letting the copter do a stable crash landing. I also look for any opportunity to drag him over to the evil side of multirotors. :slight_smile:

16

[quote=“dazzab”]Are you saying that it’s not possible to maintain thrust and control with the copter as it’s configured? The only way to maintain stability is to let it crash land? If that’s the case then Arducopter certainly is not suitable for my purposes.

I’ll bet you a hundred dollars that if I put my Wookong controller on this copter that it won’t crash when it does a RTL. Many people have the exact copter built to the same specs and their copters don’t crash land when doing RTL. So why is it unreasonable for me to expect the same with Arducopter?[/quote]

I’ll bet you a hundred dollars that I can make a Wookong controller crash by tuning its PID gains unstably. You’re blaming the platform while admitting to the multiple mistakes that caused your crash.

Because it thinks it needs that much power because you’ve overtuned it. Each motor has constraints, and at some point something has to be prioritized over something else. The only safe thing to do is to prioritize roll and pitch.

17

[quote=“jschall”]
Because it thinks it needs that much power because you’ve overtuned it. Each motor has constraints, and at some point something has to be prioritized over something else. The only safe thing to do is to prioritize roll and pitch.[/quote]
First of all, I’m not blaming anything. I’m just reporting what I see and asking questions. There’s no need to get sensitive when someone makes a comparison to another flight controller. I take your point about changing the gains on the Wookong though. That’s a fair point. It would be fun to try though wouldn’t it? It would be great to have two exact copters with the two different controllers to play with in such situations.

I understand what you have said and appreciate your help/feedback. It’s a shame we can’t get together and fly the copter together. My confusion stems from the fact that when I ‘overturned’ the PIDs I flew the copter quite aggressively to ensure that it would fly well with those gains. It did.

So why would it exhibit this problem simply by it changing in to RTL mode mode? Why would this cause it to suddenly require so much stabilisation as to cause a crash? It just doesn’t make sense. If the copter is as sensitive as you say due to the increased gains then why did it not exhibit this behaviour when I flew it aggressively to test just such a scenario? Why can’t I repeat the failure? Isn’t that a main requirement of testing/troubleshooting?

You behave as if the code can’t possibly be an issue here. That’s certainly possible but who knows, there might be an opportunity to make it more robust. I would like to remind you though that this is the exact copter that led to the discovery that the code for the barometer had an issue when it became hot. Tridge and Randy fixed that immediately. I’ve got a few ideas stemming from your feedback that I’ll discuss with them. If the code is solid, which it probably is, and can’t be improved to handle high hover rate situations then even a warning to pilots might be an idea? Anything that stops possible damage has to be a good thing, right?

I’m on my way to fly with the CUAV team right now so I’ll have a chat with them about this and see if I can get Tridge to go over the code with me. Thanks again for your time and I’ll let you know how things go after I make the changes you suggested. It will be a couple of days as I damaged a prop in that last crash and a new set will take a couple of days to get here. I’m sure lowering the gains and going to a larger prop will help. I’ve already had feedback from the manufacturer who agrees about the larger prop size. But it will be hard to test if aggressive flying doesn’t trigger the issue. Hopefully this scenario will not prop up again.

Cheers.

18

Leonard is probably the person to talk to if you want the stability code to change in any way.

The code is not an issue here - there’s simply a physical limit to what your motors can output, and you hit that limit. If the code tried to push it further, then roll, pitch and/or yaw would’ve fallen off the back.

You’re hitting multiple bad conditions here: ridiculously high hover throttle (your 20% margin goes to zero at only 36 degrees of tilt), unstable gains, yaw offset…

19

[quote=“jschall”]Leonard is probably the person to talk to if you want the stability code to change in any way.

The code is not an issue here - there’s simply a physical limit to what your motors can output, and you hit that limit. If the code tried to push it further, then roll, pitch and/or yaw would’ve fallen off the back.

You’re hitting multiple bad conditions here: ridiculously high hover throttle (your 20% margin goes to zero at only 36 degrees of tilt), unstable gains, yaw offset…[/quote]
That all sounds reasonable. Thanks for the explanation. I think there’s more to it though. You have yet to explain to me why if it’s so straight forward that I can’t repeat the failure manually. That’s a worry.

I’ve reported that the code is not working in the manner that you and Tridge have explained to me. For me at least. I’m not going to go through the code as you suggested and I’m not going to do the ‘simple algebra’ as Tridge suggested. The developers can look in to it or not. You don’t seem to think it’s an issue and that’s fine.

I’m disappointed that I’ll have to look in to alternatives but I simply can’t contribute any further time or money. If a simple mode change is going to crash the copter then it’s not suitable for my use. I’m going to move on to shooting videos instead.

Thanks again for your time and assistance.

20

Hi Dazza,

It looks like you have found significant number of problems with your setup, leaving only the explanation of the unexpected loss of power.

As John has said the important issues here are:

  • Your roll pitch tuning was too high leaving your copter on the verge of an oscillation.
  • Your copter is hovering at about 75 to 80% throttle.

The job of the roll and pitch pid loops is to keep the copter horizontal so that the thrust generated can be used to keep the copter flying. If the controller finds itself in a situation where it is losing control of roll and pitch, the motors may not be able to supply enough power to correct the problem. So the controller adjusts the throttle output to give the controllers the power they are asking for. This results in the best chance of regaining control and returning the copter to it’s commanded attitude. Once the copter is back under control the thrust can again be applied to keeping the copter up. This all happens in 1/100th of a second. The only way the throttle can remain low in this situation is if the PID loops continually ask for more power than the copter can provide at the commanded throttle setting. This is what the logs clearly show happened to you.

I hear you say, “but my copter was level, it was under control”. It looked like your copter was under control because it didn’t flip. But it was not under full control. The logs show that the copter was very quickly oscillating back and forth. And this was taking the stability patch to it’s limits to stop the copter losing control completely. To do this your throttle was being reduced because your cotper is heavily loaded.

So why did this happen then. Well first up, your gains mean that it only takes the slightest change to set your copter oscillating and that oscillation will be fast and difficult for you to see from the ground. It looks like something physically changed between the two flights on the log. This may have been a slight movement in one of the arms causing the yaw offset John mentioned (I would not be surprised if something is looser than it should be). However, what ever changed was enough to set the copter stability right on the edge. From here anything can make the difference, battery voltage, wind, airspeed, roll or pitch angle, temperature. In your case you had two instances, one in alt hold and one in RTL.

We can never say there isn’t any problem in the code that could contribute to a problem like this, we can say with complete confidence that this problem wasn’t caused by the code or the stability patch. In this case the code did exactly what it was intended to do. As a result, your copter landed with enough control to do only minor damage.

The reason we can talk with such authority about your flight, despite not being there, is because the logs have a great deal of information in them. This information is not subject to human memory or impression, it directly logs what is happening in the controller. This, combined with an intimate knowledge of the code, means there is no question about, for example, what throttle your copter is actually hovering at, just weather what you thought you were hovering at is correct.

This detailed logging feature is one of the most significant features of arducopter that is not available on other popular systems. If you learn to use it you will be able to find and correct the mistakes you are making, hopefully before they lead to a crash.

I would also suggest that you will make the people helping you feel more appreciated if you spend less time suggesting code problems, or insulting comparisons with other systems, when your mistakes are pointed out.