Quadcopter Crash on AC 3.4-rc4

I ran both 7zip file and the bin file through Virus Total and nothing was detected - https://www.virustotal.com/pt/file/595e9da03d85d55992f07b21ee5258d27e630e8ccefd25a54962985dc136fe14/analysis/1473634833/ and https://www.virustotal.com/pt/file/53c2ddfdd0cea50d936601cbbc3c196752906596256b5edfc402f4073e7b52fc/analysis/1473635155/

@Frontier please try to post the log file again.

Thanks everyone for the help.
I am trying to upload the file directly here but I am getting an error from the forum for the size of the file; even if the file is only 2300kB, the forum rejects it as bigger than 3072kB (!). I’ve uploaded the log to my Google Drive account: https://drive.google.com/file/d/0B1cKwFJ-3IEsWW5mY3JLOE90WVk/view?usp=sharing

@Thorsten: The 7-zip file is clear, it is just an archived file like .zip files. Are you sure you’ve downloaded the correct file and not something else by clicking on an advertising link? The filename is S500_PixRacer_AC34rc4_2016-09-11 10-48-34.7z. I am using ESET SmartSecurity on my Windows PC and can assure you that this file is clear. Windows defender is known for false positive reports.

@Guiboy: If you mean the WiFi add-on board of the PixRacer, it was removed before flight (part of my pre-flight test routine). The 3DR telemetry module is operating at 433MHz, so no chance of interfering with the FlySky TX/RX combo.

I have this theory: the aircraft was operating fine with rc2. When I clicked on MP for the beta firmwares, although it reported that rc4 was available - and installed it - the binary was actually rc3 which has an altitude throttle bug. The MP reports rc3 on the window titlebar and either this was a typo and it is actually rc4 or there was a mixup and rc3 is downloaded and flashed from MP besides labelling as rc4. Just a theory.

Again, thank you very much in advance for your help.
Hope the log does help as I am clueless what really happened.

@Frontier
Pretty sure about the download and the log file. But who knows :slight_smile: At least defender showed the correct name of the file. Hence my warning.

Re your log: you have a severe vibration problem. It seems something came loose inflight. The log does not contain too much data but maybe someone can find out more,

@Thorsten: If you mean the vibrations after 30" switching to PosHold, then these were caused by the uncontrollable full throttle of the quadcopter. The funny thing is that after retrieving the crashed quadcopter, all critical componens (Pixracer, RX, PDB etc) are still mounted on the S500 PCB frame, ESCs too :slight_smile:

Thank you for the insight :slight_smile:

@Frontier
At the first event (22-23), the vibes were already high before CTUN.ThI and CTUN.ThO went high.
Maybe a prop broke and then AP tried to compensate and clipping caused the full throttle.

What is strange indeed is that the vibes calm down in some cases when you switch to another flight mode.
Just a guess: do you use active braking and self-tightening props? But then the prop should have spun off…

Hi, i think i had the same problem, reported at Copter-3.4-rc4 available for beta testing

My quest started to climb right after moving the stick to around 60% in AltHold. Then I had to bring back in manual, with weird attitude changes.
Also after a few seconds of being armed, i get bad AHRS message, event after recalibrating acc/compass.
I just rebuilt the rc3, and will check hopefully today after work.

@Marcel_Kovacs: I’ve checked your issue too on the link you provided and I think I had the same issue too.

On rc2 when connected to the quad with MP via 3DR telemetry, everything was working as it should and information was instantly updated on the MP HUD.
After updating to rc4, I remember that I performed the same test again (flying around in PosHold and then invoking RTL): the quad completed the test as it should but I remember that after it landed when I tried to disconnect from MP it reported that the aircraft is still in the air!

Stupid of me, this should have be enough of a warning to revert to rc2 instead of flying with rc4, my quad would be fine by now.

Please do report here if the rc3 you built works fine, I’d like to test it when I rebuild my quad or - better still - revert to rc2 where everything worked just fine.

@Thorsten: The ESCs I have are these (40A) and they run BLHeli (I have no idea which version but the startup chime is different from BLHeli 11.1, so it should be newer). No idea about active braking either, I just plugged them to Pixracer and calibrated them all at once without issues. The day of the crash I was flying with standard non self-tightening 9450 blades (DJI clones), the day before when still flying rc2 I was using genuine DJI 9450 self-tightening blades.

I noticed this issue with MP reporting Rc3 after flashing Rc4 on the Pixracer also.

Wow… 700Mts high, what a fall.

Your fence was disabled. If it were enabled , would it be enough prevent the climb? I’m asking to learn if AC is smart enough to detect this “flyaway” condition despite the reason behind.

No rcin or out logged, unfortunately.

From 36 on, the mode was stabilize. The thO was low and the copter was still climbing. I don’t understand why

Thanks very much for testing.

So Thorsten found the issue already which is that the vibration levels are really really high. When the throttle is at full the vibration levels are around 100m/s/s ~ 120m/s/s. So that’s about 10G. Normally anything above 60m/s/s is quite bad and will lead to poor altitude hold performance.

If the Mission Planner was attached (or another GCS that shows vibration levels) I’m pretty sure it would colour the “Vibe” message red on the HUD.

Can I ask how the flight controller is mounted to the frame?

We’ve got a couple of wiki pages on mounting and how to measure vibration levels.
http://ardupilot.org/copter/docs/common-vibration-damping.html
http://ardupilot.org/copter/docs/common-measuring-vibration.html#common-measuring-vibration

So I don’t think this incident is due to a software difference between -rc2 and -rc4. It’s more likely that in previous flights the throttle level was kept low enough that the vibration levels also remained within limits.

@rmackay9: Thank you very much for taking the time to look at the log.

The AUAV Pixracer board was inside a cloned Pixracer aluminum case and the case was mounted directly to the frame using four 3M foam pads I’ve “tailored” (apparently not thick enough as nothing is provided by AUAV in terms of material to help mount the board). The foam pads kept the Pixracer construction firmly mounted to the frame PDB; imagine that even after recovering the crashed quad (which fell of 700m height), the case with the board is still mounted on the frame. Before mounting it, I’ve consulted the wiki on vibration damping but unfortunately the Pixracer is not an easy board to work with as it is very small and most ready-made damping solutions do not fit.

That’s how the board was mounted.

For the next rebuild (waiting a new S500 frame to arrive from China), I plan to use a damping platform for the CC3D evo board (which is the same size like the Pixracer): 1, 2.

Of course I am open to any better suggestions/solutions, provided that they can be shipped to Greece.

I have a similar frame and a Pixracer and I use the Orange foam from HobbyKing mounted on 3d printed carrier and my vibrations are less than 5.
http://www.hobbyking.com/hobbyking/store/_71328__Anti_Vibration_foam_Orange_Latex_190mm_x_140mm_x_6mm_AR_Warehouse.html?strSearch=orange foam
I only use 4 small squares hot glued to the frame and printed mount.

Mike

Back in apm days i had a skyrocketing fly away due to the very same reason. I’ve managed to save my copter almost by a miracle. Since then I’m a little bit traumatized by that experience. I took a very heavy shot of Adrenalin :slight_smile:

That’s why I would like to learn if fence would have been enough to mitigate the crash.

I believe a broken propeller on a hex copter would not be enough to bring it to an uncontrollable crash by the lack of thrust, however it could due to the vibrations that it might cause. Am I right?

Also, I thought that in Stabilize mode, vibrations would not cause the related behavior that frontier reported, of putting it into Stabilize mode and bringing the throttle down (the way I saved my old quad from going to the clouds).

@tabascoz: The current situation is that I cannot trust ArduCopter 3.4-rc4 because of the stabilize thing. It was quite traumatic for me not to be able to take control of the aircraft in stabilize mode, it should be possible to cut throttle instead of climbing uncontrollably. I too thought that stabilize could operate regarding of the vibration readings, as it is supposed to be a manual flight mode.

@rmackay9: I believe I had geofence enabled on rc2 (I will search my other computer, I think I have the .param file for rc2 stored there) but on the rc4 log it appears deactivated. I did not performed a full “factory reset” on the Pixracer after upgrading from rc2, could it be that I had some flash space corruption that deactivated the geofence? There is also the question @tabascoz made: could geofence helped in not letting the aircraft ascend uncontrollably?

Again, I’d like to thank anyone contributing to my issue, as I would like to learn from it as much as possible.

My guess is the flight controller is bouncing around inside the case. If not, the vibration dampening on the vehicle is not sufficient. It’s hard for me to know but undoubtedly the vibration levels are far far out of tolerance. To put it in perspective, they’re about 8x higher than my IRIS. It’s so bad that there are 33 thousand clipping events also recorded in the logs. A clipping event is where the accelerometers hit their maximum and we lose information (i.e. we lose the peak).

The fence would not have helped in this case, because although the vehicle would have tried to stop at the fence and/or switch to RTL, to stop it’s climb the altitude controller needs to be able to control it’s altitude. When the vibrations are this bad, it can’t do that.

Recovering in stabilize would have worked. In the logs the vehicle goes into stabilize a few times and it does descend.

After this event, some of the other developers and I talked about adding a fall back to a baro-based altitude hold. It’s possible but it’s not easy. I think we will add that in a future release though because safety is obviously very important and so we should do what we can.

I just want to add that with a new vehicle, it’s possible to measure the vibration levels in real time. They’re displayed on the mission planner by clicking on the Vibe message that appears on the HUD.

@rmackay9:
Thank you very much for the detailed answer.

It is good that it is confirmed stabilize could help in high vibration situations like this. Unfortunately, everything happened so fast and by the time I’ve switched to stabilize, the aircraft was already out of my sight.
Although I cannot see the board to bounce inside the case, I will also use extra double-sided foam to secure it better.

Again, thank you for considering adding the fall-back to barometer based altitude hold to ArduCopter, it really improves safety.

When the parts arrive and after rebuild, I will do more tests on rc4 (or whatever version is current by that time) and provide feedback.

@Frontier
Hi

You can track the progress of a possible solution here https://github.com/ArduPilot/ardupilot/issues/4825

As a general rule I would urge all users to reserve one channel for the Motors Emergency Stop.

http://ardupilot.org/copter/docs/channel-7-and-8-options.html

Remember that it kills the motors, BUT you have 5 seconds to change your mind :slight_smile:

@LuisVale: Thank you very much for your help and suggestions.
I will flash my PixRacer with -rc5 and when the new frame/motors arrive and start re-building, I’ll reserve one channel for the emergency motor stop function, just in case the new build tries to skyrocket again :smile:

Hi rmackay9,

     Happy to hear that you and your parter(other developers) have been aware of this issue, I have also met this issue before and resulted in several crash.
     After that, I spent lots of time and wanted to know the reason why high vibrations will cause rise even low throttle.
     Now we have found the reason and solved this issue as follows:
     In this function AC_PosControl::accel_to_throttle(), we calculate throttle out according to  measured accel_z and target accle_z. Therefore, if the vibrations too high or accel sensor is bad, the measured accel_z is unreliable. 
     I have analyzed our logs of crash, and found ratio in EKF2 equals to 0 or 100,and accel_z in imu1 or imu2 is unnormal and the bad accel_z caused rise.
     Now, we have two improvements: 
     1.add limitation in the function AC_PosControl::accel_to_throttle().

z_accel_meas = -(_ahrs.get_accel_ef_blended().z + GRAVITY_MSS) * 100.0f;
z_accel_meas = constrain_float(z_accel_meas, -_accel_z_max_meas, _accel_z_max_meas);

      2. add a function rate_to_throttle() in AC_PosControl::rate_to_accel_z()

_ekf.getIMU1Weighting(ratio);
if (ratio >= _ratio_threhold && ratio <= (1 - _ratio_threhold)) {
_last_imu_ok_time_ms = hal.scheduler->millis();
} else if (hal.scheduler->millis() - _last_imu_ok_time_ms > POSCONTROL_IMU_ERROR_TIMEOUT_MS) {
rate_to_throttle(_vel_error.z);
return;}

    That's to say, if ratio is normal, we use accel_to_throttle(), if not ,we use rate_to_throttle() to calculate throttle out according actual velocity_z and target velocity_z.
     Hope it  helpful to you, and hope also you can have better solution to release this issue.
     If any problem, contact me by email. 
     Eric Xiao           Email: qifu.xiao@outlook.com