More likely the software not suitably accomodating hardware failure.
Leading theory is the IOMCU reset. That could have been a hardware or a
software failure; there’s insufficient information to tell AFAIK. @eosbandi 's observation on the Vcc dip is a (very small!) piece of
evidence for HW issue.
tridge tested the hardware safety switch on the same commit and the
protections against mid-air arming held, so even if water did get into the
hardware safety switch it should not have had the effect you saw.
Thanks for everyone, a lot. Im starting to get, what happened. If I understand correctly, we hade a very rare problem caused by a mix of hardware and software issues. So if the first error was hardware related (water), the software should have saved the day, same as if it was an IUMCU bug. And none of this should have happened if we were using the 3.6.12 version? Or the PR by @tridge is the only way to solve this issue in the future?
Also we will update some of our older aircrafts to the 3.6.12. version anyway (the newer ones are already running on 3.6.12.), and also use the PR by tridge, right? If yes, can you help, how to deploy it? And hopefully this is my last question, thanks a lot again, Im really gratefull!
You are right, we’ll go with the latest version, thanks. And also, is it correct, that with a newer version the crash could never happen? Or the only 100% solution is the PR by @tridge? And if yes, how should we install it?
tridge’s PR has been merged to master and is marked to be backported to the 4.0.x series. That will happen in the next few days, but it will still be several weeks before that makes it out, if I’m thinking right.
We can’t provide an absolute guarantee, but I do think a repeat is very unlikely. The 3.6.10 release was the first where we used the new IOMCU firmware and we fixed several bugs since then.
The PR I did to force safety off after a IOMCU reset has gone into the Copter-4.0 branch in preparation for the 4.0.4 release.
Cheers, Tridge
I ran your log file through the Auto Analysis with Mission Planner V1.3.70 to get this information.
Mission Log print out below.
Log File C:\Users\Mike\Desktop\2020-05-08 13-01-54.log
Size (kb) 9222.1337890625
No of lines 110486
Duration 0:05:06
Vehicletype ArduCopter
Firmware Version V3.6.10
Firmware Hash 1c04a91e
Hardware Type
Free Mem 0
Skipped Lines 0
Test: Autotune = UNKNOWN - No ATUN log data
Test: Brownout = GOOD -
Test: Compass = WARN - WARN: Large compass offset params (X:238.49, Y:-234.63, Z:-30.30)
WARN: Large compass offset in MAG data (X:238.00, Y:-234.00, Z:-30.00)
Moderate change in mag_field (25.49%)
Test: Dupe Log Data = GOOD -
Test: Empty = GOOD -
Test: Event/Failsafe = FAIL - ERRs found: CRASH GPS_GLITCH
Test: GPS = UNKNOWN - join() takes exactly one argument (2 given)
Test: IMU Mismatch = GOOD - (Mismatch: 0.38, WARN: 0.75, FAIL: 1.50)
Test: Motor Balance = FAIL - Motor channel averages = [1546, 1509, 1656, 1613, 1552, 1503, 1650, 1619]
Average motor output = 1581
Difference between min and max motor averages = 153
Test: NaNs = FAIL - Found NaN in CTUN.TAlt
Found NaN in CTUN.DSAlt
Test: OpticalFlow = FAIL - FAIL: no optical flow data
Test: Parameters = FAIL - ‘THR_MIN’ not found
Test: PM = GOOD -
Test: Pitch/Roll = UNKNOWN - ‘BarAlt’
Test: Thrust = GOOD -
Test: VCC = UNKNOWN - No CURR log data
Hi there again, and thanks for all the help we got from the community, it helped us a lot to understand what happened last time. But now we had a similar crash, can you maybe take a look at the log, because it seems the same, as last time (VCC voltage glitch, and than power safety) but this time we used the 3.6.12 firmware (factory default, we can not upgrade to 4.x.x ourselfs) wich had critical bug fixes, including the solve for this problem, right? Or the only solution for this is to use the PR that @tridge implemented to the 4.0.4 release?
CubeBlack definitely needs to be on latest stable (4.0.3 at the moment) with the SB2 parameters. If you’re prevented from upgrading yourself, then get onto your supplier right away.
Make sure you use latest Mission Planner or QGC too - they’ll tell you if you’ve got a known hardware issue with the CubeBlack.
There’s something odd going on with your motor output - seems to be a weight (centre of gravity) imbalance to the right rear. I’ve just shown a selection of the outputs here for clarity.
This battery voltage is dropping a lot for only short flights, and there’s no current logging so I can’t tell if that voltage drop is to be expected. It’s best to set up current measurement and logging.
I couldnt be sure of the crash moment or what caused it - there’s many people more experienced with those details than I am.
PIDs and some other parameters looked to be near to standard - I would have thought there would be greater deviation from standard for a large octo, but maybe it’s correct. My thought is tuning could be a bit better probably, but is not really the cause of a crash or anything serious as far as I can tell.
Z vibrations are a bit higher than ideal - this might be affecting the ability to detect landing properly and disarm. Maybe look for loose items or any wiring that’s rubbing or pulling on the flight controller.
If you’re worried about the safety switch causing issues set:
BRD_SAFETYOPTION,0
You’ve already got:
BRD_SAFETYENABLE,0
Thank you very much, it was very helpful, and we are trying to get the manufacturer to upgrade asap. @xfacta can you tel me a bit more about the SB2 parameters? What should be done with them? And also, if i set the BRD_SAFETYOPTION to zero, itt will disable the safety switch? It would be very good until the manufacturer finally upgrades.
Of course i will check all the details you advised, thank you very much!
Hello ,
I had exactly the same problem IOMCU reset on an autopilot not used too much pixhawk 4 holybro,firmware( custom) 4.1 stable,the flight was low altitude controlled and protected for test, it was like the drone lost control for half a second after it took control again but I cut the motors to avoid taking any risk.
but when i saw the log i was surprise that rcouts were frozen for almost 2 seconds !, then after that weird radio failsafe and land, I asked this question in this link RCOUT freezes during the flight, IOMCU reset! no answer yet ,there is the log , but i need to find the cause because it can be dongerous if happen again , ,why the IOMCU reset !
I must specify that the firmware is custom because I adapt it to my Heli coaxial drone, then I use six servos and two esc, I believe that this IOMCU problem is caused by some kind of electrical disturbance between the outputs and the escs servos, is it possible that the pixhawk hardware is poorly electrically isolated!?
Hello everyone.
We just had a similar problem with our research drone. It crashed and fell into the water while it was carrying an even more expensive camera.
All we could get was the tlog.
We are contacting the manufacturer of the equipment, but we would really appreciate some extra opinions on the matter.
Any chance anyone could have a look?
May have been a motor burning out; current spikes as the motor dies.
I’m rather intrigued by your tlog - it’s Copter 4.0.5 but something has translated the tlog into Portugese? Is that happening on-board? In that case you’re running custom code and really have to rely on manufacturer support for assistance.
We sent it for maintenance and it came back 3-4 months ago.
The reason we sent it is that it wasn’t flying “nose-first”. They said the issue has been fixed by calibrating the RC Controller as it was interfering with the equipment during the flight (even while on auto mode) - so it kept spinning in the yaw axis. While the equipment was there, they also replaced a few parts: motor bearings, and the clamps that hold the arms open. If I remember well, they also mentioned updating the firmware. We also had to update our ground station software (which is a customised MissionPlanner).
We couldn’t fly the drone as we received it back from maintenance as there was a “critical error - no terrain” - so we had to have remote assistance to update one or some of the parameters (“drone follows the terrain”, if I remember well). I still have some previous logs and this parameter setting file, if that’s of any help to find some missing information?
As the manufacturer has certain rules about warranty, we never change anything without them assisting. For that same reason, we also don’t perform any changes to the design of the equipment.
I am afraid I won’t be able to provide you with the information about the parts as they were all customised with the manufacturer’s brand. However, we have some logs from a previous flight, if that helps. Do you think you could get this data from there?
It’s a Brazilian manufacturer so they probably translated it. I could translate it if it helps.
Just post the screenshot and I’ll translate.
These plots and your observations seem pretty helpful.
About the peak, do you mean the red followed by the blue, than followed by a negative peak in green and yellow?
Is that correct to assume that blue and red are the same type (eg. both CC) and green and yellow are the other way around (eg. both CCW)?
I noticed shortly after peaking, red and green seem to be working in sync - can I assume that they are adjacent motors?
Ia there any indication of the position of each servo?
When I plot the position of the drone in 3D it seems that the event starts with a clockwise roll:
could it mean that two motors failed at the same time - maybe those in the right? Would that make any sense?
On a side note: according to the course instructors, it should be able to fly if only one of the motor fails (but I’ve never seen it actually doing it, so I don’t know if that is really true).
So far we had to rely on them for maintenance, support, even updating some simple parameters - as they are the ones who customised and built it.
(Just in case you’re wondering - nope, I didn’t choose to buy this specific model)
So I wonder whether we would receive an impartial analysis from them in a situation where, for instance, they screwed up.
Since you have received the running code on your vehicle, you also have
the right under ArduPilot’s license to receive a copy of the source code
to whatever is running on your vehicle. Access to that source code would
actually be much more convenient than a translation of a screenshot
Motors 1 and 2 are opposite corners of the vehicle. When one motor fails that corner will drop, so ArduPilot commands the motor output to max. The other corner is going up, so ArduPilot commands its output to minimum. Roughly speaking.
(I say “motor”, but it could be anything to do with the lift on that corner. Other things that can fail are ESCs, wiring harness and physical things like the prop flying off.
Technically it is possible to fly with a motor out on a quadcopter - it involves the vehicle spinning like a top, and ArduPilot categorically does not support this. ArduCopter master will not fly a quadcopter with a motor out.
