Our 30K Drone crashed, and we have no idea why, but we got the logs, maybe somebody can help?

I ran your log file through the Auto Analysis with Mission Planner V1.3.70 to get this information.

Mission Log print out below.

Log File C:\Users\Mike\Desktop\2020-05-08 13-01-54.log
Size (kb) 9222.1337890625
No of lines 110486
Duration 0:05:06
Vehicletype ArduCopter
Firmware Version V3.6.10
Firmware Hash 1c04a91e
Hardware Type
Free Mem 0
Skipped Lines 0
Test: Autotune = UNKNOWN - No ATUN log data
Test: Brownout = GOOD -
Test: Compass = WARN - WARN: Large compass offset params (X:238.49, Y:-234.63, Z:-30.30)
WARN: Large compass offset in MAG data (X:238.00, Y:-234.00, Z:-30.00)
Moderate change in mag_field (25.49%)
Test: Dupe Log Data = GOOD -
Test: Empty = GOOD -
Test: Event/Failsafe = FAIL - ERRs found: CRASH GPS_GLITCH
Test: GPS = UNKNOWN - join() takes exactly one argument (2 given)
Test: IMU Mismatch = GOOD - (Mismatch: 0.38, WARN: 0.75, FAIL: 1.50)
Test: Motor Balance = FAIL - Motor channel averages = [1546, 1509, 1656, 1613, 1552, 1503, 1650, 1619]
Average motor output = 1581
Difference between min and max motor averages = 153
Test: NaNs = FAIL - Found NaN in CTUN.TAlt
Found NaN in CTUN.DSAlt
Test: OpticalFlow = FAIL - FAIL: no optical flow data
Test: Parameters = FAIL - ‘THR_MIN’ not found
Test: PM = GOOD -
Test: Pitch/Roll = UNKNOWN - ‘BarAlt’
Test: Thrust = GOOD -
Test: VCC = UNKNOWN - No CURR log data

YumaMike

the .Tlog file 2020-05-11 08-12-06.tlog doesn’t match the .Log file. 2020-05-08 13-01-54.log
YumaMike

Hi there again, and thanks for all the help we got from the community, it helped us a lot to understand what happened last time. But now we had a similar crash, can you maybe take a look at the log, because it seems the same, as last time (VCC voltage glitch, and than power safety) but this time we used the 3.6.12 firmware (factory default, we can not upgrade to 4.x.x ourselfs) wich had critical bug fixes, including the solve for this problem, right? Or the only solution for this is to use the PR that @tridge implemented to the 4.0.4 release?

Thanks in advance!!

hi
i want to know can you flight after this problem(IOMCU reset)?

1 Like

CubeBlack definitely needs to be on latest stable (4.0.3 at the moment) with the SB2 parameters. If you’re prevented from upgrading yourself, then get onto your supplier right away.
Make sure you use latest Mission Planner or QGC too - they’ll tell you if you’ve got a known hardware issue with the CubeBlack.

There’s something odd going on with your motor output - seems to be a weight (centre of gravity) imbalance to the right rear. I’ve just shown a selection of the outputs here for clarity.

This battery voltage is dropping a lot for only short flights, and there’s no current logging so I can’t tell if that voltage drop is to be expected. It’s best to set up current measurement and logging.

I couldnt be sure of the crash moment or what caused it - there’s many people more experienced with those details than I am.
PIDs and some other parameters looked to be near to standard - I would have thought there would be greater deviation from standard for a large octo, but maybe it’s correct. My thought is tuning could be a bit better probably, but is not really the cause of a crash or anything serious as far as I can tell.

Z vibrations are a bit higher than ideal - this might be affecting the ability to detect landing properly and disarm. Maybe look for loose items or any wiring that’s rubbing or pulling on the flight controller.
If you’re worried about the safety switch causing issues set:
BRD_SAFETYOPTION,0
You’ve already got:
BRD_SAFETYENABLE,0

Thank you very much, it was very helpful, and we are trying to get the manufacturer to upgrade asap. @xfacta can you tel me a bit more about the SB2 parameters? What should be done with them? And also, if i set the BRD_SAFETYOPTION to zero, itt will disable the safety switch? It would be very good until the manufacturer finally upgrades.
Of course i will check all the details you advised, thank you very much!

Hello ,
I had exactly the same problem IOMCU reset on an autopilot not used too much pixhawk 4 holybro,firmware( custom) 4.1 stable,the flight was low altitude controlled and protected for test, it was like the drone lost control for half a second after it took control again but I cut the motors to avoid taking any risk.
but when i saw the log i was surprise that rcouts were frozen for almost 2 seconds !, then after that weird radio failsafe and land, I asked this question in this link RCOUT freezes during the flight, IOMCU reset! no answer yet ,there is the log , but i need to find the cause because it can be dongerous if happen again , ,why the IOMCU reset !
I must specify that the firmware is custom because I adapt it to my Heli coaxial drone, then I use six servos and two esc, I believe that this IOMCU problem is caused by some kind of electrical disturbance between the outputs and the escs servos, is it possible that the pixhawk hardware is poorly electrically isolated!?

Hello everyone.
We just had a similar problem with our research drone. It crashed and fell into the water while it was carrying an even more expensive camera.

All we could get was the tlog.
We are contacting the manufacturer of the equipment, but we would really appreciate some extra opinions on the matter.
Any chance anyone could have a look?

Here’s the tlog file: https://drive.google.com/file/d/1sJy0lqZluDPcc0ywfGJ0tx9tNsos5GA_/view?usp=sharing

I understand the log is in the water together with the drone but some more details would be useful.

  • Motors
  • Esc
  • Fly time before the accident
  • Firmware
  • In case it was a new firmware how many flights on it before accident
  • Make and model of the pixhawk
  • AUW
  • Battery volts and capacity (i think 6s from the log)

The more details the better to try to understand what happened.

It’s quite clearly a motor failure:

May have been a motor burning out; current spikes as the motor dies.

I’m rather intrigued by your tlog - it’s Copter 4.0.5 but something has translated the tlog into Portugese? Is that happening on-board? In that case you’re running custom code and really have to rely on manufacturer support for assistance.

Peter

1 Like

Hi, thanks for your help

The drone is/was very similar to this one: https://mundogeo.com/wp-content/uploads/2020/11/12155550/image3.jpeg , only the gimbal was a little different and adapted to the MicaSense Altum we had)

We sent it for maintenance and it came back 3-4 months ago.
The reason we sent it is that it wasn’t flying “nose-first”. They said the issue has been fixed by calibrating the RC Controller as it was interfering with the equipment during the flight (even while on auto mode) - so it kept spinning in the yaw axis. While the equipment was there, they also replaced a few parts: motor bearings, and the clamps that hold the arms open. If I remember well, they also mentioned updating the firmware. We also had to update our ground station software (which is a customised MissionPlanner).

We couldn’t fly the drone as we received it back from maintenance as there was a “critical error - no terrain” - so we had to have remote assistance to update one or some of the parameters (“drone follows the terrain”, if I remember well). I still have some previous logs and this parameter setting file, if that’s of any help to find some missing information?
As the manufacturer has certain rules about warranty, we never change anything without them assisting. For that same reason, we also don’t perform any changes to the design of the equipment.
I am afraid I won’t be able to provide you with the information about the parts as they were all customised with the manufacturer’s brand. However, we have some logs from a previous flight, if that helps. Do you think you could get this data from there?

Thanks for your help, Peter!

It’s a Brazilian manufacturer so they probably translated it. I could translate it if it helps.
Just post the screenshot and I’ll translate.

These plots and your observations seem pretty helpful.
About the peak, do you mean the red followed by the blue, than followed by a negative peak in green and yellow?
Is that correct to assume that blue and red are the same type (eg. both CC) and green and yellow are the other way around (eg. both CCW)?
I noticed shortly after peaking, red and green seem to be working in sync - can I assume that they are adjacent motors?

Ia there any indication of the position of each servo?

When I plot the position of the drone in 3D it seems that the event starts with a clockwise roll:
could it mean that two motors failed at the same time - maybe those in the right? Would that make any sense?

On a side note: according to the course instructors, it should be able to fly if only one of the motor fails (but I’ve never seen it actually doing it, so I don’t know if that is really true).

So far we had to rely on them for maintenance, support, even updating some simple parameters - as they are the ones who customised and built it.
(Just in case you’re wondering - nope, I didn’t choose to buy this specific model)

So I wonder whether we would receive an impartial analysis from them in a situation where, for instance, they screwed up.

Therefore my post in here asking for help.

Since you have received the running code on your vehicle, you also have
the right under ArduPilot’s license to receive a copy of the source code
to whatever is running on your vehicle. Access to that source code would
actually be much more convenient than a translation of a screenshot :slight_smile:

Motors 1 and 2 are opposite corners of the vehicle. When one motor fails that corner will drop, so ArduPilot commands the motor output to max. The other corner is going up, so ArduPilot commands its output to minimum. Roughly speaking.

(I say “motor”, but it could be anything to do with the lift on that corner. Other things that can fail are ESCs, wiring harness and physical things like the prop flying off.

Technically it is possible to fly with a motor out on a quadcopter - it involves the vehicle spinning like a top, and ArduPilot categorically does not support this. ArduCopter master will not fly a quadcopter with a motor out.

If they have code inside that can do that, and you bought their product, you have the right to get the source code. It would be nice if you contributed it back to us in the form of a github pull request.

I do not think they did anything that fancy, but it would be good if they did :stuck_out_tongue:

It would be fun to add it though. link :slight_smile:

I for some reason here a script calling… I will let those with padded rooms test it though…

I will certainly try and get that from them to have a closer look.

thanks for the explanation. So, according to the tlog, 2 of the motors failed at the same time, right?
When I plotted the tlog using https://plot.ardupilot.org/ I could tell the drone Rolled clockwise and the information about the motors confirms it.

Is there anything that could help indicate or rule out any sort of collision (with a bird, for instance), prior to the abnormal behaviour of the drone?
obs: I checked some vibration parameters but I am not sure they would capture this aspect - what do you guys think?

Is there anywhere I could get this information in the documentation? I tried to find it, but maybe I didn’t use the right keywords.

I doubt they did it, either.
Who claimed that it could fly with 3 motors was one of the course instructors and unfortunately I don’t have it written down or documented. After hearing that I felt much more confident to fly the drone with our expensive camera.

that is impressive!
now that I try and picture our (RIP) drone doing it, I see a couple of issues: just to begin with, the propellers were huge and dead sharp. And as the drone was not too small either, I guess someone could’ve gone badly hurt. The size was very similar to their v2: link.

Vibration in Z seems to increase and start to drop before the others (X and Y).

Does/Could it indicate anything in particular?

The Roll seems to happen before the peak in the servo outputs. Does it mean the behaviour of the motors happened in an attempt to correct the Roll?

  • Is there any indication that the roll could have been caused by a problem affecting one side of the drone (two motors at the same time)?

  • Is there any clear indication that the motors had an electrical failure or does it only look like a physical/mechanical problem?

Any thoughts?

Hello , did you find the cause of your problem? IOMCU reset ?
I had this on one of my machines (helicopter) it disappeared but now it’s back on a new machine, but this time every takeoff for this specific machine , I take my time to understand the real reason for this problem, I changed the autopilot (pixhawk 4 ), the servos, ESC ,bec … it change nothing , I don’t believe that a drop of 0.1v causes this !!, so why not the FMU CPU restarts too !
i can be wrong but for my case I believe that there is a kind of hardware weakness ,lack of insulation ,ground loop , between autopilot and actuators can cause this .

Unfortunately, we didn’t find the drone and all we could get were some hypotheses and speculations from the analysis of the telemetry logs.