Sudden rolling causing crashes

Hi everyone,

As part of my university project I’m building a hexacopter. Please forgive the mess it is when you see the videos…

All was going well until the other day it started behaving badly. It unexpectedly rolls and can’t always re-stabilize itself. On some occasions it has caused it to flip.

We are using a pixhawk to control the craft. The craft flew fine for the first week, and although small changes occurred, nothing points to the cause of the issue.

Here is a link to the craft flying well on it’s maiden voyage: dropbox.com/s/6m3lg1t0o3mpj … 0.mov?dl=0

Now here is a video of what it’s doing: dropbox.com/s/zetwoe9a3juaz … 7.mov?dl=0

We have swapped motors, escs and cables to eliminate those; the problem still occurs.

I have included a log of the problem, although it isn’t the same one that you see in the link. One point to note is it always follows motor 2. Here is a link because of its size : dropbox.com/s/mpnltpwf372v0 … n.log?dl=0

Notice how channel 2 ramps at the same time as the roll suddenly changes. On this flight it was able to re-stabilize, but it continually occurs. I’m not sure on the order of events or the whether roll to the right is +ve of -ve?

I would include better logs but I’m having trouble getting it to log both IMU and channel PWM. I also know a video with the right log would help, but I can’t do that just yet.

What I’am asking for is help in diagnosing and interpreting the logs to hopefully solve the issue, my limited knowledge is hindering this process.

Thanks

Any questions please ask :slight_smile:

FYI a roll to the left is minus, which is what you are seeing.

The reason you see a big spike in the output to motor 2 is that your Pixhawk is trying to correct the roll to the left it is sensing. To counteract the roll it is requesting more power from motor 2 and less power from motor 1 (opposite from motor 2). But that’s not usually stopping the roll.

It’s most likely something in the power system to motor 2 is fading, perhaps due to heating. This could be the motor itself (not too likely), the ESC (a good suspect) or a poor power or servo connection. Could even be your motor 2 plug into the Pixhawk having a bad connection.

Another, less probable possibility is something is going weird with your ESC for motor 1 and it’s going uncommanded to higher power. But that would be really unusual. It’s far more likely there is an intermittent failure in your motor 2 power system.

Thanks Otherhand,

So my personal analysis was exactly the same as yours, yet I only guessed that right roll was +ve…blah blah blah. So I was reading the log correctly, that’s good :slight_smile:

So we swapped the ESC from motor 6 to motor 2 and the same with the signal wires and it occurred again on motor 2. It’s always that arm. As I mentioned, the ESC (this also includes power ports on the distribution board), motor and signal cables have all changed and been swapped about.

Here is a video of the other crash (skip to 1:00) : dropbox.com/s/xupbhy6wvd5ty … 9.MOV?dl=0

If the ESC has been changed and power ports swapped, what else could cause this rolling?

:slight_smile:

If you’ve changed out the motor and ESC already then the only plausible thing left would be to take a close, hard look at the wiring. Perhaps a bad solder joint somewhere. I’ve done that to myself a couple of times.

A longshot possibility might be something bad with the Pixhawk output 2, like a bad solder joint on the board. Not sure how you’d check for that through as your problem is intermittent.

You might want to consider tying the thing down and running up the motors to full throttle for a while and see if there’s any failing of the number 2 motor. Usually things crap out when pushed hard and your goal should be to see if you can get it to fail.

Thing is, we’ve changed the wiring from pin out (Pixhawk) all the way to the motor.

Our setup for signal is Pixhawk>Power distribution board (it has signal ports)>ESC>Motor

I even swapped motor 1 and 2 ports on the PD board and on the Pixhawk pinout (so they were still motor 1 and 2 but running through the others wiring) and the problem followed the motor 2 arm.

I’m assuming this is a weird issue that hasn’t arisen before? I’ve looked at the Pixhawk internals and excluding tiny solder cracks everything looks great.

The full throttle idea might just have to occur, although I’m still non the wiser as to why it changed between flying well to being a pain, when nothing happened to cause it…

The other idea might be to change the PD board :confused: Could a short be causing the issue? The chassis is aluminum and the PD is only crudely isolated by electrical tape.

It could be the PD board, but rather than a short, although possible, I’d look for a break.

Maybe related, maybe not, I don’t see any logging of current (it would be good to see if the current spiked). Not sure what’s up with that. Your Vcc doesn’t drop, so that’s suggestive, but not conclusive, of no short.

It sounds like you’ve been pretty thorough and have checked everything obvious, which makes this all the more a bitch to solve.

Thanks OtherHand :slight_smile:

What would logging of current look lik? I’m assuming it would look similar to the throttle curve, increasing etc as more is demanded.

We are powering the Pixhawk via the supplied power module and I have battery monitoring turned on with the settings Voltage & Current, Power Module and Pixhawk. The GS appears to give good information with percentage decreasing over the time we fly and voltage somewhat similar, current moves around as we fly, seems to correlate with the flight pattern.

I have plenty of flights with this occurring (we have been troubleshooting like crazy) so I may just have a flight where current is logged :slight_smile:

You’re right, this is a complete pain! Ill check the wires in the PD board too to try and eliminate that.

One point, could a vibration in the axis cause this? Im not sure that log include IMU data. It’s difficult to determine cause from consequence in the logs so I’m not sure if the vibrations felt are as result of the roll or cause of the roll and the unit is trying to overcome “weird” readings.

I’ve just read some posts saying that the ESC ground needs to be connected to the Pixhawk as well as the signal line, could this be an issue?

Current logging would generally follow your throttle plot. However a short might show a large current spike. One drawback is current isn’t logged at a rate as often as some parameters so a very short spike could be missed. It’s odds that it’s not showing up in your log, as from what you described it should be logging. I’ll look again at the setting recorded in your log.

Vibration data is recorded in the IMU fields in your logs. In looking at the log you posted there are some spikes in vibration, but they occur immediately AFTER your undesired roll events. That’s probably due to its trying to regain level orientation. Note that the vibration levels are recorded in the Pixhawk, which is presumably isolated a bit from the frame. You could have frame vibrations affecting a break on your PDB, happening only if the frame hits a specific resonance.

With a Pixhawk you really need to run the ground from each ESC back to the Pixhawk. With an APM you could get away with just running the signal wires but a Pixhawk is more temperamental in that sense. Not running ESC grounds have caused many problems for people. Usually it’s more systemic, affecting different motors and looking like a loss of motor sync, but you are seeing it happen on the same motor/channel. Still, this is a very likely culprit.

Hey, it’s not a “university project” unless you have to do some thinking…

I’ve now run the ground to the Pixhawk too. I haven’t been able to test yet as finding space was difficult in Friday.

I’ve also tried to isolate the PDB a little to minimise vibrations.

I will update after testing :slight_smile:

It’s definitely causing me to do some thinking!

Thanks

So tested the ground theory. No luck.

Next on the list is test the copter in the + configuration; this will either move the orientation of the roll or keep it the same. if it remains the same then it will most likely point to something with the chassis, otherwise I blame the Pixhawk.

Thanks for looking.

Further help or suggestions would be great.

You could try reflashing the Pixhawk’s firmware to something older, like 3.1.5 and see if the problem persists. It sounds like you’ve been able to rule out problems in your power distribution, so if it still acts up with older firmware, then it’s possibly a bad Pixhawk.

Can you point me in the direction of how to manually flash the Pixhawk? I have only used the Mission Planner method and that always uses the most up to date version I think.

From Googling it appears I may have to use the command line, surely there is an easier way?

Thanks OtherHand :slight_smile:

On the Mission Planner “Install Firmware” page there is an option at the lower right to “Pick Previous Firmware”. Just select that one and it will give you a bunch of older choices.

So frame type changed from hex X to hex +, 3.1.5 tried and Pixhawk re-orientation tested, both cause sudden rolling issue. The roll has moved to arm 4 of the + config (it used to be arm 2 of the X config).

This time I have both log and video of the event:

LOG: dropbox.com/s/tzz4xog7u2v3a … n.log?dl=0

VIDEO (skip to 00:50 for rolling): dropbox.com/s/xw5tsznv3zmm4 … 7.m4v?dl=0

Unfortunately not all parameters are logged :frowning:

Looks like a new Pixhawk, need to get onto the supplier…

Thoughts?

OK, I’m really puzzled now. With the Hex in + configuration the pitch down/roll right behavior is consistent with a failure of motor#4. Now that would be the #4 output on the Pixhawk, And previously, in X configuration, the failure was in motor #2 (thus Pixhawk output #2). Sooooo…That means it’s not an issue with your Pixhawk’s output. If it were a faulty #2 output you wouldn’t have seen the problem move to #4. This suggests to me your Pixhawk is OK.

So it would seem the problem HAS to be between the Pixhawk and the motor. It sure seems like it’s an issue with an ESC, but you say you’ve switched them out. I notice the problem doesn’t appear until it’s flown a bit, which is consistent with some sort of intermittent, heating failure effect. You haven’t mentioned if you’ve tied the thing down and run it at full throttle to try and induce a failure. Might be worth a try.

It’s not clear to me if the problem is occurring on the same arm as earlier. When changing from + to X I wouldn’t expect the question arm to move from 2 to 4. Did it?

You probably should have your logging bitmask set to include all the RCOUT info (i.e., the motors) so you can see what signal are being sent to the motors. But I suspect the answer will be the same as earlier.

You know, you could try doing some crazy testing like disconnect two motors, reconfig the others and run it as a quad X. It should still fly.

Cheers OtherHand,

So the problem arm has moved. It was #2 on the Hex X config and is now #4 in the + config (180 degrees across from the old #2). The Pixhawk was rotated 30 degrees clockwise while the chassis stayed stationary, meaning the rolling has moved from #2 to #1 in the X config. Odd ey? It has moved with the Pixhawk…

I haven’t done the tied down test yet as I need to create something to hold it down. Not ruling it out, just need to get onto it :slight_smile:

My bet was on the ESC before we swapped them around and the fact it has now moved to a different arm (after frame type change) with what we thought was a working ESC makes it even more confusing.

We have done some custom wiring to a couple of the ESCs to ensure they reach the motors. At first we believed it was this causing the issue, however after swapping them around and the problem never changing arm suggested it wasn’t this wiring. But now I’m going to go back and check the wiring to ensure they aren’t the cause. I sure hope it isn’t this as it’ll have been a waste of your time…

After checking the wires I will look into doing a full throttle test to push it to the limit. My thoughts would be that if it doesn’t cut out then we are back at looking at the Pixhawk?

I will also try a non custom ESC on the problem arm and see if it remains an issue or moves (I doubt it will as it didn’t when we tried it last time).

Testing in Quadcopter format and excluding the problem arm might be a good way of testing the Pixhawk theory…

#4 is using an ESC with standard wiring, just checked.

Just did the full throttle test, at half throttle and I got some interesting results.

I have both video, picture and logs.

I’ll upload asap after I’ve looked at them :slight_smile:

Just thought you’d wanna know :slight_smile:

Test was carried out in Hex + config

Video of the flight can be found here: dropbox.com/s/pnwym2nrn7ue1 … 1.m4v?dl=0

Log for the video is here: dropbox.com/s/4vbzlaf7fyebl … n.log?dl=0

Log with RCOU is here (not related to video but just afterwards): dropbox.com/s/4vbzlaf7fyebl … n.log?dl=0

http://planner.ardupilot.com/wp-content/uploads/sites/5/2013/12/hexa-octo.jpg

Watch for motor #3 (watch the sticker) and notice it slows down then speeds up, the slow mo part towards the end might help you notice it. Motor #3 used to be motor #2 in the old X config (see link).

My theory is that due to a hardware issue motor #3 slows down and causes a roll/pitch to the left which is caught by motor #4 slowing down, when motor #3 speeds back up it causes the sudden roll/pitch in the other direction that motor #4 has slowed down to compensate for, this throws it off balance and causes the rolling on arm #4.

So perhaps it’s always been arm #3? Thoughts?

Motor #3 is using the custom wiring ESC (just an ESC extension cable).

This test was not done on full throttle because of issue arising at lower setting.