Switch to Loiter and then crash

Log Analysis -- deprecated Copter Logs -- deprecated
1

After a lot of testing of 3.1.2-rc2 with a heavy lift coax octo I finally started testing with my gimbal and camera rather than a dummy load. This afternoon I was flying in Alt-Hold and when I switched to Loiter the copter started rolling back and forth about 60% each way and was out of control. It also started to drop rapidly. I switched to stabilise mode and gave it full throttle to no avail.

My gimbal was significantly damaged but fortunately my camera survived. I was flying over a grassy field and it’s been raining all day so the ground was soft. The copter was about 15 meters hight at the time and just a few minutes in to the flight. I’m a bit surprised as I had a very good flight at the same location with it earlier in the day. I’ve had a look at the log but all I can see is a confirmation of what happened. I’m not sure what to look for to see what caused this.

I would really appreciate some help figuring this out. The same copter dropped out of the sky a couple of weeks ago when I switched it out of RTL to Stabilise. I suspect in that case one of the two battery connectors was loose so I didn’t have enough current to recover but I’m not sure.

At this point I’m really worried about flying Arducopter with this copter. It’s meant to be set up for video but I can’t get it anywhere close to as smooth and stable as that task requires even though it been flying reasonable. I also noticed in my earlier flight today it’s doing the same yaw wiggle I reported with my hexa. All in all I don’t have a good feeling about all this.

I wasn’t able to get a flash log to download from the Pixhawk so I’ve attached the TLog instead.

2

I managed to get the flash log to download by renaming the existing log directory so it would be recreated. I’ve attached it to this message in case it’s of any help analysing what caused this crash. Hope it’s the correct log as I just selected the last one.

Cheers.

3

You can always take the sd card out and get the .bin log files off the card

4

Wow, a lot can happen in only 3 minutes!
Roll, Pitch and Yaw (and therefore the altitude) is very very symmetric after time=28000!

5

Thanks, that worked well. I’ve attached the file 11.bin which loads in to the log viewer fine.

You can clearly see that when I switch in to Loiter that the RCin for Roll and the RCout for Roll do not match. Also you can see the channel 3 (throttle) clearly shows me inputting full throttle but the RCout for channel 3 is all over the place?

Is this interference or possible a bad transmitter/receiver?

6

Please read this

copter.ardupilot.com/wiki/common … sing-logs/

and see if you have one of the common failures.

7

[quote=“Craig3DR”]Please read this
copter.ardupilot.com/wiki/common … sing-logs/
and see if you have one of the common failures.[/quote]

That was one of the first things I did. There is an indication that the GPS Dhop increased and the number of satellites dropped by one. But i wouldn’t think that would cause a crash. If it does, then the platform is in big trouble.

I’m well known to the developers. I’m trying to contribute to Arducopter by running it on a large professional system that most people certainly wouldn’t be willing to risk. I have hundreds of hours of testing on numerous airframes and a significant financial investment behind this contribution.

I’ve now had two unexplained crashes that involved the simple action of just changing from one mode to the other. I don’t know if this is related to the most recent release but I’m a bit surprised that the developers haven’t shown more interest given the possibilities. After all this is the copter that discovered the firmware shipped with the Pixhawk gave very inaccurate barometer results once it became hot.

At this point I won’t be flying with the Pixhawk until this situation is properly analysed which will unfortunately require more experience than I have. I had a $3,000 camera on it during this flight which I only did after a week of flying with dummy pay load to ensure things were stable. The camera was not damaged but my $800 gimbal is in several pieces and it cost me a few hundred dollars to repair the landing gear on the copter. I’m sure you can understand that unless I can work out how to get some reliability then I’ll have to fly this with a commercial flight controller and forgo all the innovation that the Pixhawk/Arducopter platform provides. That is a very disturbing prospect to me given how fantastic this platform is.

8

Hi Darrell

I’ll get Jonathan to have a look at your log and we’ll get you sorted out.

9

Thanks, that worked well. I’ve attached the file 11.bin which loads in to the log viewer fine.

You can clearly see that when I switch in to Loiter that the RCin for Roll and the RCout for Roll do not match. Also you can see the channel 3 (throttle) clearly shows me inputting full throttle but the RCout for channel 3 is all over the place?

Is this interference or possible a bad transmitter/receiver?[/quote]

RCOU are your individual motors - so channel 3 is motor 3, not throttle. So far there’s nothing obvious - I will keep looking at your log. It is definitely trying to recover. Our control scheme prioritizes roll/pitch rate demands over throttle demands - it is really averaging about half throttle when it is attempting to recover from the roll error, which results in the descent since your hover throttle is around 650 (plus the 60deg roll). Your demanded throttle is 100%, but it’s really getting 50% because the copter is trying so hard to level itself.

It may just be a matter of your control gains. 8.625 on STB_ gains seems like it would be unstable on a big octo like that. I’ve attached a graph of demanded vs achieved roll rate, and you can see the way it gets bad at the end. The only other thing I can think of that would cause this is a hardware failure… Notice that the curvature of the green line (X gyro) goes opposite what it should be right after the first peak. To me that says “something failed here”

I can work on a tool that will tell me what the roll, pitch, and yaw impulses from the motors should be based on RCOU. Then I can try to correlate that to the gyro and see if an obvious failure pops out.

edited to include attachment.

10

[quote=“Craig3DR”]Hi Darrell
I’ll get Jonathan to have a look at your log and we’ll get you sorted out.[/quote]
Thanks for that. I’m stuck on this and really need to work it out so I appreciate the assistance. Having someone else look that has experience is also a very good opportunity to learn.

BTW, I’ve moved my questions here as it was mentioned a few times on DIYDrones that support questions should be posted here. But I notice the developers aren’t very active here and that Randy still deals with most issues on the DIY forum. Because I’m running the new versions on a larger copter I was hoping to contribute my feedback. But I have a feeling it’s not getting through.

I’m assuming that this forum is designed to take the burden off the developers but in my case it’s not working out that well. I think in this case Robert L might be interested as I’m suspecting the root cause of this may be the auto tuning going over board a bit on the PIDs. ???

11

That’s very interesting and useful info. Thanks so much for taking a look. I too was suspicious of those high PIDs but I’ve done a few auto tuning sessions now and that’s what it seems to come up with. And it flies very well in general. It was moving the same direction in a breeze when this happened. I was wondering if the wind just blew it over and it just couldn’t handle the momentum of the speed and being tilted at the same time. So maybe it’s just a case of the gains being too high? Unfortunately I’d prefer not to test that scenario again. :slight_smile:

The copter does have a 2.4Kg payload underneath it so there’s a fair amount of weight underneath it. I don’t know if that works in it’s favour or against it in this situation.

I saw the graph you included and that’s what prompted me to submit this log. Sure looked like something bad happened.

Thanks again for your time.

12

99% of user issues do not need Randy or Tridge to answer them. It is not a good investment in their time to be spending 4-6 hours per day answering support questions when they could be writing code and there are other people who can answer questions. If there is an issue that comes up that is a flight code issue then we will get Randy or Tridge involved but this forum is designed to allow more people to answer support questions and it makes it easier to track issues without having 60 page blog posts with 3000 posts in them. Eventually we will get everybody weened off of DIYDrones for support.

13

That’s almost word for word what I said to Tridge a few months ago. I actually manage a computer support service for the not for profit sector and I am a Unix Sys Admin so I have a fair amount of experience in the area. At the time he disagreed with us saying that it was important for the developers to have feedback and keep in touch. I guess in an ideal world they would have 48hrs a day but of course that’s not possible.

So I agree with you 100%. But it isn’t working out that way at all. On DIYDrones the developers specifically ask for feedback which they will need to do their work. I’m in a unique situation so wanted to make sure I provided them with feedback. Although as you said now I can’t even find the threads in the hundreds of pages of replies. Nonetheless, that’s how the baro temperature issue was discovered. When I provided feedback on DIYDrones I received assistance quickly. Here, it’s days if at all. So maybe there aren’t others who can answer the questions just yet. I try to contribute but let’s face it, the people who code it are the people who know it the best.

Yes, I know it’s early days. I am NOT complaining but rather giving you my feedback. What’s happening here is quite amazing really. It’s a tremendous adventure.

It’s a very odd situation as the software is open source and 3DR is commercial. I see 3DR as the hardware side of things and Randy/Tridge/Lenoard/Robert the FOSS side of things. Then again, Randy is an employee right? So as you can see it’s all a bit messy for a user to decide where to be. A post on DIYDrones, or maybe here, or maybe on the developers mailing list? Heck, recently I was told to put a feature request up on Git rather than here. Right, I’m going to set up a Git account just to make a feature request. I don’t think so.

Good luck with that. Let me know if I can help in any way.

14

Can you describe how you tuned the copter?

I suspect Autotune isn’t coming up with gains that are too high for your copter, I think the problem is you are flying the copter in a different configuration to the one you used to autotune.
If you are flying with a large gimbal that is free to move on the frame you will probably find that auto tune doesn’t work. If you do autotune with a load firmly attached to your copter then replace it with a gimbal your parameters will probably be too high and verging on unstable.
The best approach to using autotune in this situation is to remove the gimbal and do autotune with the lightest setup you can (that doesn’t cause the battery voltage to sag). This should give more conservative pids that will be safer with your gimbal.
However, the final performance depends on the resonate frequency of the gimbal and suspension system. Autotune is not designed to tune a copter with this extra degree of freedom. In the end you must ensure the copter is stable yourself and make manual adjustments if needed.

15

[quote=“Leonardthall”]Can you describe how you tuned the copter?
I suspect Autotune isn’t coming up with gains that are too high for your copter, I think the problem is you are flying the copter in a different configuration to the one you used to autotune.
[/quote]

Well, now you tell me! :slight_smile: I have been posting about this and even mentioned it to Tridge as I wasn’t sure of the best procedure to use when using a gimbal. I’ve seen other users mention that as well. Sure wish this would have come up earlier as I’ve had two bad crashes and waited weeks for support. Oh well, better late than never I guess.

[quote=“Leonardthall”]
If you are flying with a large gimbal that is free to move on the frame you will probably find that auto tune doesn’t work. If you do autotune with a load firmly attached to your copter then replace it with a gimbal your parameters will probably be too high and verging on unstable.[/quote]

Well, there you go. I thought the gimbal moving around would cause issues so I replaced the gimbal with a dummy load that doesn’t move around too much. It moves a bit but I thought that would be a good representations. So I did give it a fair amount of thought.

[quote=“Leonardthall”]
The best approach to using autotune in this situation is to remove the gimbal and do autotune with the lightest setup you can (that doesn’t cause the battery voltage to sag). This should give more conservative pids that will be safer with your gimbal.[/quote]

Well, that’s the info I needed. Thanks. But it certainly doesn’t seem logical. I would have thought that you want the copter in the configuration you will be flying with during tuning which is why I specifically didn’t tune it with the gimbal attached. I hope this info gets documented somewhere where others will see it so they don’t have to learn the hard way as well.

[quote=“Leonardthall”]
However, the final performance depends on the resonate frequency of the gimbal and suspension system. Autotune is not designed to tune a copter with this extra degree of freedom. In the end you must ensure the copter is stable yourself and make manual adjustments if needed.[/quote]

Good info, thanks. It’s probably a good idea to publish this disclaimer in the documentation don’t you think? How does a person know what a reasonable tune is and what isn’t? The copter flew fine after auto tuning even with the high PIDs. I tested it with dummy load for a week before putting the expensive gear on it. I also tested auto tuning on smaller copters and got great results which gave me a lot of faith in the results it comes up with.

What more can a user do? The issue only showed up after a second event where the copter fell after a mode switch. Of course, the issue seemed like switching modes and not related to tuning. It was only after Craig looked at the logs that things came together. After all, you don’t zip around with a copter like this so the event that showed the issue won’t show up immediately.

I’m very disappointed here. I received little to no help when I reported the first incident. One developer wasn’t interested in teaching me how to figure out the results, and then here I had to validate myself and the situation to get someone from 3DR to look in to it. It was only by accident you came here and provided assistance after I mentioned it on DIYDrones. But users have specifically been asked not to post support requests there.

I’m trying really hard to contribute and to be supportive. But it’s getting harder and harder. Given how much money I’ve spent to do so I’m beginning to wonder why I bother.

But kudos to you. Your code is fantastic and I really appreciate your assistance.

16

Glad to help.
If the rest of the dev’s are anything like me they are still getting used to the new forums.

17

[quote=“dazzab”]
I’m very disappointed here. I received little to no help when I reported the first incident. One developer wasn’t interested in teaching me how to figure out the results, and then here I had to validate myself and the situation to get someone from 3DR to look in to it. It was only by accident you came here and provided assistance after I mentioned it on DIYDrones. But users have specifically been asked not to post support requests there.[/quote]

Hi Darrell, I hope you don’t mean me. I am more than willing to help you - would you like to chat on skype or g+ sometime? I’ll PM you some more direct ways to contact me.

Last I heard, Leonard is going to add acceleration demand limiting to the stab controller. I think that would have prevented this problem.

18

[quote=“Leonardthall”]Can you describe how you tuned the copter?
I suspect Autotune isn’t coming up with gains that are too high for your copter, I think the problem is you are flying the copter in a different configuration to the one you used to auto tune.
[/quote]

I’m not convinced Leonard. I’ve autotuned a quad a hexa and an octo and all three came up with gains well over 9. The quad runs an APM, the hexa and octo use Pixhawk. All have powerful Tiger Motors and T-motor carbon fibre props.

Honestly, I think it flew better on the defaults so I turned down the high 9 values to 7.

Not sure how to get the log to you. It’s 8Mb and the limit is 5. I zipped it up and now the forum says the limit for .zip file is 256k. Tried putting it on Droneshare but they only take .tlog. Sigh.

Hope this helps.

19

The Stab P value that auto-tune generates is the highest you should go. As I have said on the forums, back it off to achieve the feel you like. For camera ships I would expect this to come down to 5 or 6 to get a soft reaction to disturbances.

Can you tell us exactly how you are tuning your copter? Pictures are also good.

20

[quote=“Leonardthall”]
Can you tell us exactly how you are tuning your copter? Pictures are also good.[/quote]
Nothing special really. I take the copter up to about 10 meters or so and put it in Alt Hold mode. When it’s nice and stable I just flick the switch to start the auto tuning.

Here are photos of the Droidworx CX-600 quad and the Droidworx XM6 hexa. The gimbal and pool noodles were not on the hexa when I auto tuned it.

[attachment=1]CX600.JPG[/attachment]

[attachment=0]XM6.JPG[/attachment]