Quadcopter Crash on AC 3.4-rc4

Hello,

I am using Arducopter 3.4-rc4 on an AUAV PixRacer board installed on a (clone) S500 frame.
The quadcopter uses a uBlox Neo m8n GPS unit and everything is powered via a Matek v3.1 PDB. Current and voltage monitoring is provided by a genuine 3DR Power Module v1.0 from my old APM 2.6 quad. Motors are SkyRC X2830-950kV with 40A BLHeli ESCs. Powered by a 5200 Zop Power 30C 4S LiPo. Telemetry is a 3DR 433MHz module (clone) and the TX is a Flysky FS-i6s with a iA10 RX.

First tests (10 days ago) of this quadcopter were done using AC 3.4-rc2 and were successful: with all pre-arm checks enabled, I was able to arm using both compasses (external and internal, no compass variance errors, correct heading e.t.c) and the basic flight modes were fine (stabilize, AltHold and PosHold). Just doing simple movements - mostly testing hovering as this was intended to be a camera quad - no problems. Two times I’ve even used RTL and it came back fine without any issues.

The reason I upgraded from 3.4-rc2 to 3.4-rc4 was because I had some issues connecting the PixRacer to Mission Planner via USB cable (9 out of 10 times it did not connect at all), so when I read the announcement for rc3 fixing the issue I performed the upgrade via Mission Planner. MP reported that AC 3.4-rc4 was available in beta but after flashing the firmware, it was reported as rc3. Unfortunately, I did not noticed that yesterday after performing the upgrade as I thought it was just a mismatched version string. After upgrading, I recalibrated the radio, the accelerometer and the compass. No issues.

Today, I went to a football field up to a mountain nearby to test the system. Everything went fine initially: performed compass calibration with no issues (both compasses working, external is #1), took off in stabilize mode and then switched to AltHold, no issues. Immediately I switched to PosHold and the S500 was responding just fine: holded position perfectly besides some small gusts (1-2 Beaufort). I then proceeded in flying the quadcopter around the field at about 10m from the ground.

While I was flying around in PosHold, I was simply trying to get the ‘feel’ of this new firmware version by doing what I was doing in the past days (unfortunately, there are no datalogs from rc2 as I had a corrupt FAT32 filesystem on the PixRacer MicroSD card and had to format it); doing quick passes in PosHold trying to simulate camera recording. Telemetry on the ground was OK, I was using Tower Beta and it had correct readings for battery, height and flight mode (PosHold with 3D+DGPS).

While flying around suddenly the quadcopter begun to rise without using throttle - the stick was at the middle position - at full speed and in a couple of seconds it was at about 90m high. Immediately I flipped the RTL switch to bring it back but it kept climbing and could not see it anymore; I could hear the motors having full throttle but I could not see the aircraft. Telemetry on the ground did not report anything and it was stuck on PosHold mode, even if the TX switch was at RTL.

To make things short, I was in panic mode and tried everything to get the quadcopter to the ground, switching also to altitude hold and lowering the throttle at 40% so that it would come down but it didn’t. Not even switching to stabilize helped with throttle at 0%, the motors were still working in full speed and I could not see the aircraft. In a minute or two (cannot remember) the quadcopter fell to the ground about 30 meters outside the football field and crashed onto rocks. The area is not occupied so none was hurt. As for the quad, broke two arms, the landing gear, two motors are totalled, the ESCs might be OK but can’t be sure (their LEDs work and I can hear the motor tunes the two other motors do), the GPS is OK but the pole is broken and the battery is OK as well (I found the quad nose down between some rocks and it took almost 1 1/2 hour to locate it). PixRacer and the RX are working too and was able to locate the SD card so I have the log of that flight.

I am attaching the flight data flash log here in hope that someone could help me determine what went wrong. Before the quad ‘got mad’ everything seems OK and vibrations are acceptable; after that point they go to extreme levels.

Many thanks in advance for any help.

WARNING!

The zip file seems to contain malware - at least that’s what Windows Defender reported.

Best regards,
Thorsten

Hello
Are you using wifi link + RC around 2.1 and 2.4 GHz?
It already occured to me that the drone was just flying very well but at a certain point of time the wifi just took over the RC in a manner the RX did not detect the Failsafe (because there was constant signal recording for throttle… which was wrong coming from wifi interference). Everything I did on my RC was effectless (even shutdown).
Fortunately I had set a geofence that activated RTL after the fence breach.
Be aware Wifi should be set at 5 GHz if working together with RC link to avoid interferences.
I hope this helps
Guillaume

Please re-post the log file natively to the site and no via a third party site, and make sure it been virus checked. Thx

I ran both 7zip file and the bin file through Virus Total and nothing was detected - https://www.virustotal.com/pt/file/595e9da03d85d55992f07b21ee5258d27e630e8ccefd25a54962985dc136fe14/analysis/1473634833/ and https://www.virustotal.com/pt/file/53c2ddfdd0cea50d936601cbbc3c196752906596256b5edfc402f4073e7b52fc/analysis/1473635155/

@Frontier please try to post the log file again.

Thanks everyone for the help.
I am trying to upload the file directly here but I am getting an error from the forum for the size of the file; even if the file is only 2300kB, the forum rejects it as bigger than 3072kB (!). I’ve uploaded the log to my Google Drive account: https://drive.google.com/file/d/0B1cKwFJ-3IEsWW5mY3JLOE90WVk/view?usp=sharing

@Thorsten: The 7-zip file is clear, it is just an archived file like .zip files. Are you sure you’ve downloaded the correct file and not something else by clicking on an advertising link? The filename is S500_PixRacer_AC34rc4_2016-09-11 10-48-34.7z. I am using ESET SmartSecurity on my Windows PC and can assure you that this file is clear. Windows defender is known for false positive reports.

@Guiboy: If you mean the WiFi add-on board of the PixRacer, it was removed before flight (part of my pre-flight test routine). The 3DR telemetry module is operating at 433MHz, so no chance of interfering with the FlySky TX/RX combo.

I have this theory: the aircraft was operating fine with rc2. When I clicked on MP for the beta firmwares, although it reported that rc4 was available - and installed it - the binary was actually rc3 which has an altitude throttle bug. The MP reports rc3 on the window titlebar and either this was a typo and it is actually rc4 or there was a mixup and rc3 is downloaded and flashed from MP besides labelling as rc4. Just a theory.

Again, thank you very much in advance for your help.
Hope the log does help as I am clueless what really happened.

@Frontier
Pretty sure about the download and the log file. But who knows :slight_smile: At least defender showed the correct name of the file. Hence my warning.

Re your log: you have a severe vibration problem. It seems something came loose inflight. The log does not contain too much data but maybe someone can find out more,

@Thorsten: If you mean the vibrations after 30" switching to PosHold, then these were caused by the uncontrollable full throttle of the quadcopter. The funny thing is that after retrieving the crashed quadcopter, all critical componens (Pixracer, RX, PDB etc) are still mounted on the S500 PCB frame, ESCs too :slight_smile:

Thank you for the insight :slight_smile:

@Frontier
At the first event (22-23), the vibes were already high before CTUN.ThI and CTUN.ThO went high.
Maybe a prop broke and then AP tried to compensate and clipping caused the full throttle.

What is strange indeed is that the vibes calm down in some cases when you switch to another flight mode.
Just a guess: do you use active braking and self-tightening props? But then the prop should have spun off…

Hi, i think i had the same problem, reported at Copter-3.4-rc4 available for beta testing

My quest started to climb right after moving the stick to around 60% in AltHold. Then I had to bring back in manual, with weird attitude changes.
Also after a few seconds of being armed, i get bad AHRS message, event after recalibrating acc/compass.
I just rebuilt the rc3, and will check hopefully today after work.

@Marcel_Kovacs: I’ve checked your issue too on the link you provided and I think I had the same issue too.

On rc2 when connected to the quad with MP via 3DR telemetry, everything was working as it should and information was instantly updated on the MP HUD.
After updating to rc4, I remember that I performed the same test again (flying around in PosHold and then invoking RTL): the quad completed the test as it should but I remember that after it landed when I tried to disconnect from MP it reported that the aircraft is still in the air!

Stupid of me, this should have be enough of a warning to revert to rc2 instead of flying with rc4, my quad would be fine by now.

Please do report here if the rc3 you built works fine, I’d like to test it when I rebuild my quad or - better still - revert to rc2 where everything worked just fine.

@Thorsten: The ESCs I have are these (40A) and they run BLHeli (I have no idea which version but the startup chime is different from BLHeli 11.1, so it should be newer). No idea about active braking either, I just plugged them to Pixracer and calibrated them all at once without issues. The day of the crash I was flying with standard non self-tightening 9450 blades (DJI clones), the day before when still flying rc2 I was using genuine DJI 9450 self-tightening blades.

I noticed this issue with MP reporting Rc3 after flashing Rc4 on the Pixracer also.

Wow… 700Mts high, what a fall.

Your fence was disabled. If it were enabled , would it be enough prevent the climb? I’m asking to learn if AC is smart enough to detect this “flyaway” condition despite the reason behind.

No rcin or out logged, unfortunately.

From 36 on, the mode was stabilize. The thO was low and the copter was still climbing. I don’t understand why

Thanks very much for testing.

So Thorsten found the issue already which is that the vibration levels are really really high. When the throttle is at full the vibration levels are around 100m/s/s ~ 120m/s/s. So that’s about 10G. Normally anything above 60m/s/s is quite bad and will lead to poor altitude hold performance.

If the Mission Planner was attached (or another GCS that shows vibration levels) I’m pretty sure it would colour the “Vibe” message red on the HUD.

Can I ask how the flight controller is mounted to the frame?

We’ve got a couple of wiki pages on mounting and how to measure vibration levels.
http://ardupilot.org/copter/docs/common-vibration-damping.html
http://ardupilot.org/copter/docs/common-measuring-vibration.html#common-measuring-vibration

So I don’t think this incident is due to a software difference between -rc2 and -rc4. It’s more likely that in previous flights the throttle level was kept low enough that the vibration levels also remained within limits.

@rmackay9: Thank you very much for taking the time to look at the log.

The AUAV Pixracer board was inside a cloned Pixracer aluminum case and the case was mounted directly to the frame using four 3M foam pads I’ve “tailored” (apparently not thick enough as nothing is provided by AUAV in terms of material to help mount the board). The foam pads kept the Pixracer construction firmly mounted to the frame PDB; imagine that even after recovering the crashed quad (which fell of 700m height), the case with the board is still mounted on the frame. Before mounting it, I’ve consulted the wiki on vibration damping but unfortunately the Pixracer is not an easy board to work with as it is very small and most ready-made damping solutions do not fit.

That’s how the board was mounted.

For the next rebuild (waiting a new S500 frame to arrive from China), I plan to use a damping platform for the CC3D evo board (which is the same size like the Pixracer): 1, 2.

Of course I am open to any better suggestions/solutions, provided that they can be shipped to Greece.

I have a similar frame and a Pixracer and I use the Orange foam from HobbyKing mounted on 3d printed carrier and my vibrations are less than 5.
http://www.hobbyking.com/hobbyking/store/_71328__Anti_Vibration_foam_Orange_Latex_190mm_x_140mm_x_6mm_AR_Warehouse.html?strSearch=orange foam
I only use 4 small squares hot glued to the frame and printed mount.

Mike

Back in apm days i had a skyrocketing fly away due to the very same reason. I’ve managed to save my copter almost by a miracle. Since then I’m a little bit traumatized by that experience. I took a very heavy shot of Adrenalin :slight_smile:

That’s why I would like to learn if fence would have been enough to mitigate the crash.

I believe a broken propeller on a hex copter would not be enough to bring it to an uncontrollable crash by the lack of thrust, however it could due to the vibrations that it might cause. Am I right?

Also, I thought that in Stabilize mode, vibrations would not cause the related behavior that frontier reported, of putting it into Stabilize mode and bringing the throttle down (the way I saved my old quad from going to the clouds).

@tabascoz: The current situation is that I cannot trust ArduCopter 3.4-rc4 because of the stabilize thing. It was quite traumatic for me not to be able to take control of the aircraft in stabilize mode, it should be possible to cut throttle instead of climbing uncontrollably. I too thought that stabilize could operate regarding of the vibration readings, as it is supposed to be a manual flight mode.

@rmackay9: I believe I had geofence enabled on rc2 (I will search my other computer, I think I have the .param file for rc2 stored there) but on the rc4 log it appears deactivated. I did not performed a full “factory reset” on the Pixracer after upgrading from rc2, could it be that I had some flash space corruption that deactivated the geofence? There is also the question @tabascoz made: could geofence helped in not letting the aircraft ascend uncontrollably?

Again, I’d like to thank anyone contributing to my issue, as I would like to learn from it as much as possible.

My guess is the flight controller is bouncing around inside the case. If not, the vibration dampening on the vehicle is not sufficient. It’s hard for me to know but undoubtedly the vibration levels are far far out of tolerance. To put it in perspective, they’re about 8x higher than my IRIS. It’s so bad that there are 33 thousand clipping events also recorded in the logs. A clipping event is where the accelerometers hit their maximum and we lose information (i.e. we lose the peak).

The fence would not have helped in this case, because although the vehicle would have tried to stop at the fence and/or switch to RTL, to stop it’s climb the altitude controller needs to be able to control it’s altitude. When the vibrations are this bad, it can’t do that.

Recovering in stabilize would have worked. In the logs the vehicle goes into stabilize a few times and it does descend.

After this event, some of the other developers and I talked about adding a fall back to a baro-based altitude hold. It’s possible but it’s not easy. I think we will add that in a future release though because safety is obviously very important and so we should do what we can.

I just want to add that with a new vehicle, it’s possible to measure the vibration levels in real time. They’re displayed on the mission planner by clicking on the Vibe message that appears on the HUD.