Pixhawk crash went nuts during landing appr, have logs/video

Log Analysis -- deprecated Copter Logs -- deprecated
1

Hi guys, well took my Tarot T960 with Pixhawk for a fly today…

had about 5 good flights without issue, at the end of my last flight, (no, batt not flat, had 3.85V per cell remaining)… it abruptly yawed left approx 90*, pitched fwd approx 45* (which became a 45* right roll once the yaw was complete) and powered up heading for the hills,

Thankfully it responded to me shutting the throttle, hitting the ground sideways, (minor damage)

I’m not real good on deciphering these log files, so if someone from 3DR could take a look for me??
when I run the Tlog it appears to be about 94% through the log, you can see where the HDG goes from NW to SW, but I can’t find any reason in the log as to why it happened.

It has left me with no confidence in these flight controllers at all… they go great for a while, you think you’ve got it all sorted out, but when it goes nuts, BOY, DOES IT GO NUTS!

Tarot T960
Pixhawk running 3.1.2
Futaba 10CHG
Tarot 5008 340KV Motors
NanoTech 6s 8000 2P

here is the video: youtu.be/opckzeZefkE

2

I see DesPitch+Pitch , and Roll+DesRoll follow each other fine until you take down ThrIn.
Bumpy, but it still has authority.

The bumpy ride was most likely you descending into your own propwash, the very low KV does not exactly help there, you overreacted and shut it down.

To avoid this phenomenon, descend slower, or have some airspeed speed when descending faster.
Or: the problem might have been reduced if the motors worked harder/at higher RPM (add ballast if you plan to fly with heavy payload.)

Finally: You say that you “lost confidence in these controllers” - I think “the controllers” are equally unimpressed by you - expecting them to bypass laws of physics with funny setup and flypath. :slight_smile:

3

ok, now that you mentioned yaw:
you - do not log yaw, nor do you log Mag , but if you plot yaw against ThrOut - you will see that when throut have less ripple, (more steady current, less powerful current spikes) - then your yaw tend to fall, (until it goes below 0 and rolls over to 360).

I suspect you also have the current influence the magnetometers.
To be sure, you can enable MAG logging, and hover, then hit the throttle and climb rapidly, , then do the same… you your mag values change with throttle, you know.

If your mag detected the change as you descended, 3 of the motors would reduce the throttle even more to yaw, that too would make the descend thru propwash worse.

4

Andre, thanks for looking, but no, I didn’t over-react… I’ve been flying 3D helis for 25yrs, when you learn to do tick-tocks 3’ above the ground, you learn to react to the model very quickly… mate, it yawed 90* to the left and rolled over all by itself, UNCOMMANDED you could hear the motors powering up as it did this…

and I had done the same landing 5 times just prior to this one… without any such effect… and as you can see I WAS travelling forward during the appr

what I’m wondering now, is if one ESC went out of sync… and the resulting yaw/pitch-roll was the Pixhawk trying to fight the lack of thrust on one motor, which of course will not show up on the log.

it’s a shame there isn’t more time in the log between the UNCOMMANDED yaw, and me closing the throttle… but it looked UGLY! I was right next to a busy road… and it was heading for it!!! throttle---- close!

and it was in Stabilize mode… so why would the Mag be influencing my outputs???

Cheers

BOB

5

GPS, Magnetometer is still used a little bit in Stabilize, I think it’s too little to cause any trouble anyway, but AFAIK theyre used to prevent gyro drift and compensate for g forces during acceleration/dec…)

Ok, I did not ment to say “you overreacted and put it to the ground” - what I meant was, (if you did not cut throttle, but just flew for a little while, it would most likely stabilize - because there is no indication of BIG trouble keeping desired attitude.)

Tha said, you did not log voltage, or - the battery voltage is unrealistic : going from 4.5v to 4.3 maybe, just maybe you had low battery too, that further changed the PID’s effect.

Finally, there’s no log for RCIN +RCOUT, and IMU so I don’t know what you two actually tried to do. :slight_smile:
From the data I see, it was attempting to hold proper attitude all the time. Pointing to a setup/configuration error, not something APM did on it’s own, trust me, there are plenty of good UAS that fly very reliably.

-I suggest you fix voltage logging.
-Do some mag logging and tests , if nothing else, because it’s the think people most often fail to discover until they “need” RTL. - And no, I do not think it caused this, it may have influenced yaw, but not be the cause.
-Check and recheck for sync errors, on such low KV motors, I think sync error is less visible in logs as they are faster to regain the needed RPM.

Finally, if you dare to test, I suspect this is a effect of dropping battery voltage, you have been flying 18 minutes, with 3-axis gimbal.

6

Andre, you don’t try to hover and stabilize the model when it is in a hard-over left yaw, pitching and rolling and heading for a busy road… you dump it…

I did calibrate my power module, but for some reason, the pixhawk won’t remember the settings… so Voltage is useless, a bit like RSSI… the model only flew for about 12 mins, and the gimbal has separate power… Would be very surprised if Batt was the cause… could have done whole mission again…

the more I think about it, the more I think its a Sync issue, kind of explains the yaw-pitch-roll if one motor shutdown as I am entering the flare… will have to strap it to bench and test it… as it’s a Hexa “X” if motor #5 stopped, (front right) that would (if my thinking is correct) cause a left yaw, nose fwd pitch becoming a right roll… and I wouldn’t be the first one on here with Sync issues…

Maybe this applies to me too: diydrones.com/forum/topics/pixha … ync-issues

Thanks for your input though… where did you get info that GPS/MAG is used in Stabilize mode?

BOB

7

If you are using mission planner, it had a bug that requred you to change field before values wee written. - If it ain’t fixed, that might be the reason to your voltage calib. problem.

Of course, I forgot the road, that seems to be at good distance, but of course, it’s your call, and it’s better to ditch than hit something/somebody.

The mag/compass being alway used a but, were mentioned by one of the core developers in this or the old forum, a user had problems in stab, due to reversal of the gps/mag combo.

I have no reason to believe it’s the cause of your problems. the yaw change was just an observation.
Yes, check carefully for sync, and , maybe low voltage could make your PID’s be … incorerrect at that voltage.

8

No I don’t think it has anything to do with it either… when I get home I’ll try to do a motor sync test…

Had a look through my logs… I didn’t realise that by default it actually doesn’t log that much!!!

Is that just me??? or are we suppose to activate logging of other parameters??

Anyway all I can find of interest, as most data fields are blank is a divergence between roll and desroll, and at the same time a desyaw which is followed by yaw.

If anyone who knows how to read logs well can shed some light on this, would be much appreciated…

Cheers

BOB

9

Looks like I’m not alone…

viewtopic.php?f=80&t=5995&p=13511#p13511

Looks just like mine…

and…

viewtopic.php?f=21&t=6093&start=10

diydrones.com/forum/topics/pixha … ync-issues

and people are having out of sync issues on Pixhawk that they cannot recreate on APM2.5 or direct to Rx…

So is the Pixhawk (due to it’s higher processing power) able to send a higher rate of change of PWM to the ESC and cause it to go “outside limits” ???

I mean yeh 400Hz on APM2.5 is fine, but it might not be actually changing the PWM setting that quickly…

do we have to slow the rate of change of motor output on Pixhawk??

Cheers

BOB

10

Stil same problem for me, want to reopen my topic or set it to unsolved,
Did the ground wire workaround, went ok for 2 lipos en then suddenly flip AGAIN.
Hope official 3DR wil respond or your or my sync error topics.
Chase to APM same hardware no problem and back to APM wthout any changes sync errors.
See post nr 4 in this topic
viewtopic.php?f=80&t=6445

11

Hi Peter, yeh seems to be narrowed down to be a pixhawk issue…

I hope the USA Monday morning comes soon, and someone from 3DR can shed some light on the issue!!!

Otherwise it’s a very pretty paperweight!

Bob

12

SO…

I’m wondering if this has something to do with it…

Quoting from: groups.google.com/forum/?hl=en& … R4g9sz9ssE

john...@gmail.com
Mar 24 (5 hours ago)

Is there any difference in internal update of the PWM generator? While the hardware outputs PWM at 400/490hz the internal update of the output value is something else.
I am just speculating that if the PX updates more often, this could be seen as a more aggressive motor control by the ESC and lead to sync problems not seen on APM. It could also be something as simple as different gains, leading to more/less aggressive motor behavior.

Regardless, as long as someone can confirm PX pulses are fine, this is probably just yet another motor sync problem and technically not APM related. The PX/APM is just exposing something marginal.

  • JAB

  • show quoted text -
    [u][b]
    Randy Mackay
    Mar 24 (3 hours ago)

JAB,
Ah, the internal update rate is also faster. So the motor rate is updated at 400hz on the Pixhawk vs 100hz on the APM2.

-Randy[/b][/u]

13

400 vs 100 is the internal update, external pwm is 400hz vs 490hz
possible fix could be to set RC_SPEED = 490 like the apm setting.
Can’t test this because I went back to apm until there is a fix a root cause analyzed

One similar problem I found, the sync errors all all with HK afro 30a esc’s
the served me very well with the apm but with the pixhawk ???

14

Peter, does that not mean that if an APM is INTERNALLY updating at 100hz… Then even though it sends out a PWM at 490hz… It’s only changing the PWM at a rate of 100hz,

Whereas the Pixhawk is changing the PWM at 400hz

Therefore making the APM throttle output changes smoother, and not causing the sync issue that the four times faster pix hawk does???

Am I on the right path here???

3DR Tech help??.?? Hello??.??

15

Next Question is: If I change the RC_SPEED Param to say 400, or even 100 on a Pixhawk… is that Parameter used during the Motorsync test? or is it just sent flat-out at 480Hz bypassing your user settings???

Think I might go back to flying heli’s… with a Rx to servo connection at 50Hz… was good enough for hard 3D… why do we need 400Hz anyway??? the props can’t change RPM at 400Hz!!

The Support from 3DR is notable by its absence…

Crickets…

I think I hear Crickets…

16

Bob, have you tried the tests I did in my videos regarding this issue (youtube.com/user/xoltri)? It really helped narrow down my problem and may help thecause if you can record and post similar results. I have watched your video but in my opinion it’s not conclusive proof of a sync issue. I myself had a few crashes before I managed to isolate the problem as per my youtube videos. In the end I’m hoping that the issue I had was a unique one and that my replacement will fix the problem, but if it in fact is some inherent problem with the Pixhawk the more solid evidence that can be presented the better chance there is of coming to a solution.

Same to you peter, post some video!

The devs are discussing the issue here as you pointed out I believe https://groups.google.com/forum/?hl=en&fromgroups=#!topic/drones-discuss/aR4g9sz9ssE. But I think without more direct evidence there is not much to go on. I’m not sure who at 3DR will get my Pixhawk I sent back for RMA but hopefully something comes out of that. According to tracking it should be delivered to them today.

17

Bob
I’m currently acquiring parts to put together a quadcopter. Pixhawk and t-motors MN4010 475kv 24 pole plus bits and pieces already purchased. I’ve been looking at which speed controllers to get. I initially thought this would be straight forward. During my research I’ve come across synch problems especially with high pole motors. In a number of threads it seems the flow is that the apm/pixhawk is telling the speed controller to boost/reduce power on the motors but the motors are not responding resulting in a crash. So does this point towards an error at the speed control or motor? Rather than Pixhawk.
I’m going to try the Esc32 speed controllers (I’ve ordered esc32 clones that are coming over from the US) which are 32 bit and can be calibrated using qground control to prevent current spikes and limit the overall amps that can be used. The attached link explains it in full: autoquad.org/wiki/wiki/esc32/esc … libration/ . For apm the pwm_min_period needs to be changed to 2000. For 24 pole motors I’ve come across suggestions that the esc will need a higher advance of 22 degrees. It does seem that this esc is highly configurable.
I don’t know whether this may be the root of your problem.
Jon

18

Hi Xoltri and Jon22…

OK, so yes xoltri I have done some motorsync tests… all with quite confusing/conflicting results…

If I use throttle hold as you did, in Stabilize mode, I can’t get it to go out of Sync, which is what happened inflight, so thats weird…

when I use the Terminal Motorsync test… It randomly DOES go out of Sync… mostly on motor #5…

WHICH IS what I theorised was the motor that failed… (front right on a “X” Hexa) in my crash…

re boot out of terminal, and can’t fail it in Stabilize

so I swapped out the #5 ESC for a handy spare… and all good… no longer fails during motorsync test…

So, I must have had a dicky ESC on #5??? or did it get pushed a bit too hard??

They are 480Hz ESC’s, and Pixhawk was set at 480Hz, so no prob there…

I am heading to flying field tomorrow to test it, (low and close) and we’ll see how we go, hopefully it was just a dicky ESC for me…

Thanks 3DR for all your help on this matter, NOT!

They say DJI has bad customer service… what’s worse than nothing???

If it goes nuts tomorrow, you won’t hear back from me, I’ll see you over at the Wookong forum, as I will take to all my 3DR stuff with a hammer… I’m over it, you get to a point in your 3DR career, where you just want a product that works… and am over re-inventing the wheel every time I go for a fly… I know we’re on the bleeding edge of technology, blah blah blah, More time flying, less time in the 101010101010101’s

Happy Landings

BOB

PS Andre-K, it had nothing to do with my MAG, more to do with you wanting to get your post number up

19

We are a day further, what is the verdict about the esc change?

20

Hi Peter…

I wasn’t going to come back to this site, but since you asked… IT CRASHED!!

first of all, thanks 3DR for moving my thread into Log files, Tuning, that’s the most you’ve done for me so far… albeit not what I wanted… maybe they’ll delete it next???

Anyway, so did another last minute Motorsync test, no problems, so lets go flying…

First of all, in Loiter, hovering nicely for about 1minute, then a sudden right roll… I caught it by switching back to Stabilize, where I regained control, then about another minute later, I noticed it becoming “nervous” is the only word I can think of to explain it… like the PIDS were being slowly increased or something, so thought prudent to land… about 1 foot off the ground hard right roll again, I slammed throttle shut, resulting in akward landing on right skid and right motor, with right prop in the dirt… no damage…

So I decided that was that, WAY TOO SCARY TO FLY AGAIN!!!

then a mate of mine who’s very techo also arrived at the field, and he wanted to see for himself, so we took everything off the model to avoid $$$ damage like camera, gimbal, etc etc…

so now I’m hovering in STAB, ALTHOLD and LOITER OK, except LOITER and ALTHLD has strange hunting in the vertical, but not in STAB, not sure why… it doesn’t change Altitude, just hunting coming from motors…

so back to STAB for a landing, and again about 1foot off the ground, hard pitch up, may have rolled a bit right, I again slammed throttle shut… but this time she went beyond 90* hit ground hard and rolled over… upside down… crash protection was good enough to announce on my laptop “CRASH” “Disarm” I thought oh good you know your upside down, wonderful… minor damage…

I just noticed this thread: viewtopic.php?f=80&t=6925

where Jschall says: Seems like a bug to me. We’re in the middle of completely redoing pretty much everything at the moment, so I would just wait on the next release.

That’s nice isn’t it… which led me to thinking… why are we using APM2.5 firmware anyway??? shouldn’t this great new bit of kit be afforded it’s own custom firmware??

So, to answer your question Pete, My Pixhawk has been decommissioned… we were thinking I could take it off, and put it on a little quad just to see if we can isolate the problem, but I just don’t have time for this crap… (I’m Moving overseas in a few weeks) so I’ll just shelve it till V4.1.2 comes out…

I did activate logging of nearly all, and there are some very interesting spikes in motor outputs right before I shut the throttle… but I can’t correlate them to any reason why… it’s hard to compare inputs and outputs in the log viewer…

I have the log if you’re interested…

has anyone tried an A2???

Cheers

BOB