Pls, help analyzing hexacopter crash, with log and video

Hi.

After updating to copter 3.6.0 with Chibios on an hexacopter with Pixhawk:
MSG, 80447492, ArduCopter V3.6.0 (2eedcf56)
MSG, 80447515, ChibiOS: ff603d11
MSG, 80447559, fmuv3 00430019 33375113 31313537
shortly after starting a mission (mode Auto), the drone flew away without control when descending towards a land waypoint, accelerated heavily and crashed.

The drone was being filmed from an external camera, and had also an onboard camera. The external camera video is this:

The log is this:
Copter_20181101_14.bin

The fundamental problem is that a failure is clearly noticed on the video, but the first error apearing in the log (GPS glith) happens 5 seconds later. I don’t see the origin of the failure in the log.

The sequence of events which can be observed in the video are:
-Drone armed; mode Stabilize.
-24": mode changed to Auto (mission starts; yaw changed, pointing to ROI).
-33": failure. The drone descends normally towards a land waypoint, but being one meter above ground there is some kind of failure, height oscillates, the drone is accelerated heavily, and goes away without control, being stopped by a fence.
-41": the drone hits the fence. The GPS is disconnected (GPS logs stop, but not others); it was found 10m away.

So the most simple way to relate video and log is the instant of the last GPS message, which should correspond to second 41 in the video. It is:
87152 12:09:44.381 110893205
8" before should be the problem located, moreless at 12:09:36 102893205, around line 63312.

So the main events on both video and log are:
-Auto at 24" (35270 12:09:26.730 MODE 93242650 Auto).
-Failure at video 33" (moreless 63312 12:09:36 102893205).
-GPS Glitch (73569 12:09:39.948 MSG 106460143 GPS Glitch), 3.5" after the failure.
-Land mode change not obeyed (75624 12:09:40.448 106960070 Land), 4.4" after the failure.
-Moreless 4" after the failure RCOUT.C3 has a high value during 4.5". The drone, without control crashes.
-The crash is seen at second 41 in the video. The GPS is disconnected and thrown away (last GPS parameters at 87152 12:09:44.381 110893205).
-Log continues, so it seems that the electronics has been working all the time.

I can’t find anything going wrong around the instant of the failure as appears in the video. The first error message is a GPS glitch:
73568 12:09:39.948 ERR 106460126
73569 12:09:39.948 MSG 106460143 GPS Glitch
so that happens 4" after the failure is noticed in the video, and must be a consequence of other type of failure.

This is a graphic of ATT DesPitch and Pitch:


DesPitch starts to grow anormally around:
69437 12:09:38.248 104760026
so that happens 2" after the failure is noticed in the video. Something must have failed before.

This is a graphic with other events:


The white dotted line corresponds moreless to the failure noticed in the video.
GPS.Spd starts to grow abnormally at moreless:
72757 12:09:39.401 105913163
so that happens 3.4" after the failure is noticed in the video.

The first glitch (12m) in RFNDDist1 happens moreless at:
71283 12:09:39.043 105555302
so that happens 3.4" after the failure is noticed in the video, and is probably due because it is a sonar, and probably the drone, out of control, points to a distant point.

So up to now, something unknown to me has failed, and has measurable consequences. There is a Mode message change to Land:
75624 12:09:40.448 106960070 Land
so that happens 4" after the failure is noticed in the video, but the drone does not land.

This is a graphic of RCIN.C3 and RCOUT.C3:


so noticing the failure I pulled throttle to a minimum at
69679 12:09:38.648 105160350 1090
so that happens 2.6" after the failure is noticed in the video, but was useless since mode was Auto.
Instead, it is observed that from moreless
73178 12:09:39.848 RCOU 106360457 1858
to
86616 12:09:44.248 RCOU 110760243 1888
the drone is accelerated heavily making thing go much worse, and that happens during moreless 4.5", as can be checked in the video. The drone finally crashes. Not only the drone does not land, but being out of control is heavily accelerated.

The GPS is an RTK one. GPS.Status is 5 (RTK Float) at the crash. It goes to 4 during some instants, starting at:
72757 12:09:39.401 105913163
so that happens 3.4" after the failure is noticed in the video, coincident with the RFND.Dist1 glitch mentioned before, so can be explained because the antenna didn’t point up during some instants, and can’t be the cause of the failure.

Questions:
-What can be the primary cause of the failure observed at second 33 in the video?
-Why does not the drone land and stays out of control?
-How can it be that RCOUT.C3 after the failure can have high values and cause a crash?
-What can be done to prevent such a failure?
-Am I missing something

The video on the on board camera is:


but does not give much information, more than the motors noise at the failure.

Something wrong with the build, your CW motors are always working harder.

You did not start with a full battery, and the voltage drops a lot while it is trying to fly after the initial damage.

The vibrations were never really good, but got way too high after the initial crash and the properllers probably got damaged.

You ask Something wrong with the build, your CW motors are always working harder.

You did not start with a full battery, and the voltage drops a lot while it is trying to fly after the initial damage.

The vibrations were never really good, but got way too high after the initial crash and the properllers probably got damaged.

Answers:
-most likely something did fell apart
-why should it land ? - you were flying a test in AUTO and poshold , not manual throttle modes.
-RCOUT dependd on attitude, it is working as expected given the setup/situation
-do not fly tests in autothrottle modes, stick to stabilize until you know all parts are well mounted.
-…

Thanks for your patience analyzing the logs. I am more used to use Mission Planner, but I´ll have a look at your way.

Anyhow, please observe first the videos:
http://youtube.com/watch?v=2E7Z96ppuHk (external: somethings goes wrong at second 33 while descending (~line 63312 in log))
http://youtube.com/watch?v=Gvwfjh8QWp0 (on board)
The problem is that nothing appears wrong in the log around that line; it does around 3" or 10000 lines later, when the drone is uncontrolled.

The drone had been flying with this configuration (without gimbal and camera) during months, with beta versions. This is a video four days before (with gimbal and camera) with same mission, and version:
MSG, 429266473, ArduCopter V3.6.0-rc12 (014022de)
MSG, 429266496, ChibiOS: ff603d11
MSG, 429266540, fmuv3 00430019 33375113 31313537


and this is the log:
Copter_20181028_circles-ref_4.bin

CW motors are always working harder
Observe in video above that mission include a ROI, so drone is rotated (in full video CW an CCW).

You did not start with a full battery, and the voltage drops a lot while it is trying to fly after the initial damage
This is the battery voltage:


and this is the current:

After the failure observed in second 33 of the video the drone is fully accelerated, the current is measured up to 70A so the voltage drops (out of battery terminals goes down to 12.5V), but all that is a consequence of an unknown previous failure.

The vibrations were never really good, but got way too high after the initial crash and the properllers probably got damaged
These are the vibrations:


Of course, if there is a failure (observed in second 33 of the video), the drone is fully accelerated, hits ground, a tree and a fence, there are going to be vibrations and the propellers will break, but all that happens after the failure.

why should it land?
When the drone hits the fence (8" after the failure is observed in second 33 of the video) the GPS is thrown away, there is a GPS glitch and a land command that is not obeyed.

RCOUT dependd on attitude, it is working as expected given the setup/situation
Here are all RCOUT’s:


After the failure observed in second 33 of the video, RCOUT’s oscillate (this can also be seen in the video), the drone flies away, is fully accelerated, and the battery current is measured at 70A. Is this something to be expected?

do not fly tests in autothrottle modes, stick to stabilize until you know all parts are well mounted
As said, this drone flew smoothly and reliably (many missions) during months, and four days before with gimbal and camera and V3.6.0-rc12. Changing to V3.6.0 final was the only big change.

most likely something did fell apart
That is not clear in the videos. As seen in second 33 of the external video, after a normal descend, RCOUT’s oscillate, the drone flies away and, worse, is fully accelerated.
This is the detail of RCOUT’s oscillation:


Imagining one propeller braking it seems RCOUT’s would not do this. Moreover, I remember time ago a propeller breakage while flying and the drone retained control.
Something happened, but what?

RCOUT 1 is the requested pwm to motor 1


RCOUT 6 is the requested pwm to motor 6
So if one or more is topping out , that motor is not producing the required thrust.

If the CW motor are working hard , means ccw motors aren’t straight up, causing a yaw, which is countered by cw motors to keep heading.
Fly in a hower for some minutes and graph RCOUT s ,you can see cw & ccw motors will be grouped into 2.check the motor mounts of the low pwm motors.

Again: there is a ROI.

The failure appears clearly when all RCOUT’s oscillate, as seen in the detail image above.

I think the underlying issue is perhaps ground effect pulling the barometer altitude off. This led to the altitude estimate not being correct so the vehicle flew too low and hit the ground.

Form the pic below we can see the barometer altitude drops by about 3 meters.

I think enabling the ground effect compensation will help (this feature will be enabled by default in Copter-3.7). If the problem persists if very low altitude flying is required, it could be a good idea to attach a lidar and then enable terrain following which pretty simply involves setting the Altitude Type drop-down on the Flight plan page from “Relative” to “Terrain”.

Thanks for the indication about GND_EFFECT_COMP. I didn’t know about it and will have it enabled in the future.

In fact, even with the barometer surrounded by foam inside a Pixhawk, I have always seen BARO.Alt offsets and negative values when taking off and landing. Here BARO.Alt is shown along RFND.Dist1 (TFmini) on a 10’ mission (same drone taking off and landing five days before, with V3.6.0-rc12):


so I’ll observe if GND_EFFECT_COMP=1 changes this.

That said:
-The barometer altitude going to -3m happens around log lines 14000 and 20000. The RCOUT’s oscillations (first indication of failure) happen around line 63000. Line rate is roughly 3000 lines/second.
-For low altitudes there is Benewake TFmini sonar. These are BARO.Alt and RFND.Dist1:


The TFmini gave good indications all the time, even it is seen pointing far away.
-Altitude related parameters:
EK2_ALT_SOURCE=0 (barometer)
EK2_RNG_USE_HGT=70 (EK2_RNG_USE_HGT: range finder switch height percentage)
RNGFND_MAX_CM=700, so under 490cm the TFmini should be the primary height source, and it shows reasonable values all the time.

So still I don’t see the reason for the crash, and less for why the drone is fully accelerated (RCOUT’s around 1900 during several seconds until crashing).

…the vehicle flew too low and hit the ground
It was much worse: after hitting the ground it was fully accelerated and crashed (see video).

About protection for failures, this happened recently on another drone with V3.6.2-rc2 and also a TFmini:


Initially MODE=16 (PosHold), later MODE=3 (Auto) and later MODE=6 (RTL). I then see that it is climbing without control towards the selenites and switch to MODE=16 (PosHold) for landing. I then observe that I had forgotten to connect the TFmini cable (RFND.Dist1 is 0, with no serial data) and this was not indicated at all. RTL_ALT=0, so it should return at current altitude. So an obvious error without any indication led to a dangerous situation.

Yes, ground effect and it’s influence on the barometer is a real problem especially for frames with short legs.

Once the vehicle hits the ground and tumbles it will try very hard to right itself. That could include increasing some or all motors to full power.

I’ve seen so many user incorrectly setting either EK2_ALT_SOURCE and/or EK2_USE_RNG_HGT when attempting terrain following that I’ve spent some time today fixing up the terrain following wiki pages (manual modes, autonomous modes) including adding a warning to not set these two parameters.

Re the last issue with the vehicle “climbing without control”, I’ll need to see a log…

…ground effect and it’s influence on the barometer is a real problem especially for frames with short legs

Here is a fast test of GND_EFFECT_COMP on a small quad (350g without battery, 165mm motor to motor, 5030 propellers (short legs), lifting it at home around 2.5m several times (no GPS)).

GND_EFFECT_COMP=0
Copter_20181121_34_GNDEF0.bin

GND_EFFECT_COMP=1
Copter_20181121_35_GNDEF1.bin

BARO.Alt improves.

Once the vehicle hits the ground and tumbles it will try very hard to right itself. That could include increasing some or all motors to full power
That sounds correct, so this could be the sequence of events:
-Ground effect alters descend (video second 33).
-Vehicle tumbles.
-Vehicle tries to right itself and applies full power.
However in the vicinity of RCOUT’s oscillation (start at log line 66000 moreless) the barometer signal BARO.Alt seems good. It seemed altered around lines 14000 and 20000, several seconds before.
But being so low applying full power during several seconds is an incorrect decision. Perhaps it would be better to apply to one or other motor during short intervals (such as 1") and see if it is upright.

…incorrectly setting either EK2_ALT_SOURCE and/or EK2_USE_RNG_HGT when attempting terrain following…
Parameters:
TERRAIN_ENABLE 0
TERRAIN_FOLLOW 0
(Two .DAT files in µSD TERRAIN directory)
EK2_ALT_SOURCE 0 (barometer)
EK2_RNG_USE_HGT 70 (questionable)
About EK2_RNG_USE_HGT, from http://ardupilot.org/copter/docs/parameters.html#ek2-rng-use-hgt-range-finder-switch-height-percentage
Set to -1 when EK2_ALT_SOURCE is not set to range finder. This is not for terrain following.
What is wrong?
Being TFmini data correct, how can I assure it will be used near ground?

“…vehicle ‘climbing without control’, I’ll need to see a log…”
It is the same small drone as the above GND_EFFECT_COMP tests.
Copter_20181119_30.bin
RFND.Dist1=0 all the time, but there is no TFmini serial data since its cable was not connected. Seems to think that it is really at altitude=0 and has to climb, instead of seing the obvious error (no serial data).