Quad with ArduCopter 4.3.5 falls out of the sky - any ideas are highly appreciated

New frame, fails on third flight, while still under tests. Firmware is 4.3.5, radio is herelink2, folding propellers ( MF1806 Folding Propeller), Antigravity MN5008 KV340 Motor, 6s setup (2 x 7Ah LiPo) Orange cube. Logs stop abruptly while the copter is still in the air, seemingly indicating that ArduCopter just stopped mid-air. Visually, the quad just rolled (about 360 apparently) and plummeted for 10m, breaking an arm (maybe more, at least a 20mm carbon fiber arm). I stared at the dataflash log (link here)
for quite a bit, and the only thing obviously wrong are vibrations. Those are very clearly off, and the test pilot at the time (not me) said that he observed a “lean” (to the left) indicating the the Kalman filters may be bothered by vibrations. I’m not sure what caused the vibrations (maybe the propellers, maybe something on the frame), but either way, they should not just shut down the autopilot. Any ideas on what to investigate are appreciated: we have three more identical frames, and we’d like to keep those in one piece :-).

Thank you so much for your attention,
Mihai

if logs stop abruptly, maybe the vibrations caused the power cable to the flight controller to disconnect. was power still supplied to the controller after crash?

X and Y axis vibrations are definitely a big issue. This is like prop balance, or folding prop mechanisms going bad.
This could conceivably have caused the battery to come loose and disconnect or some similar failure, like a broken solder joint.

For when you get it flying again

Check you battery monitor settings too. Somewhere in there you’ll need
BATT_ARM_VOLT,22.10
BATT_CRT_VOLT,21.00
BATT_LOW_VOLT,21.60
BATT_FS_CRT_ACT,1
BATT_FS_LOW_ACT,2 or 3
It’s a common mistake to think you dont need to set these during testing - this is probably the time you need it the most.

Try MOT_PWM_TYPE,6 as there’s no real benefit to using DSHOT1200
Lower DSHOT rates are generally more reliable and noise-immune, but some ESCs dont seem to like DSHOT150, and even some dont list DSHOT300 in the specs these days. DSHOT600 seems to work with everything, and doesnt have too many overheads in the ardupilot loops.

You can set these
INS_ACCEL_FILTER,10
INS_HNTCH_ENABLE,1 // set this then refresh params to see the rest
INS_HNTCH_MODE,1
INS_HNTCH_REF,0.14
INS_HNTCH_FREQ,50
INS_HNTCH_BW,25
INS_HNTCH_FM_RAT,0.7
INS_LOG_BAT_MASK,1
INS_LOG_BAT_OPT,4

Do a test flight in AltHold with some gentle pitch and roll movements. Let’s see that log and adjust the HNOTCH and PIDs from there. You might be able to move on to Autotune.

1 Like

The battery is not it, not directly at least: first, the batteries themselves are two of them in parallel, feeding in a custom PCB that then has two voltage regulators feeding the cube that mounts directly to the PCB. There is no wire that can be disconnected. Regarding the battery failsafe, the batteries were fully charged (if you look at the battery voltage monitor you’ll see them being above 24V), and they are good for flying for 45 minutes or so. I think that the total flight time on that set of batteries was below 10 minutes (possibly below 5 minutes). Surely, surely not the issue. What is possible (but very unlikely due to the redundancy in batteries and voltage regulators) is that vibrations shook some solder joint lose. Very unlikely though. Thank you for the ideas, if you have more, please bring them as I’m stumped!

If it would have been a wrong command to the motors (or wrong driver), I’d expect to see a panic on EKF, warning messages, and rolling on the frame, and dropping on the altitude: nothing like that in the log. A few things are really different on this frame from a previous version that flew many happy hours over the past year:

  • herlink2 has been upgraded to new firmware (now spews out two sets of non-sensical MAVLink messages)
  • ardupilot has been upgraded to new firmware (from 4.1.x or 4.0.x to 4.3.5)
  • propellers went from non-folding 17" to folding 18".
    I don’t think anything significant changed otherwise.

Thanks,
Mihai

One more thing: I’m not sure if there was still power after the crash - probably yes, but I was a few tens of meters away from the crash. When the pilot came with it, the batteries were disconnected. However, after reconnecting the battery, the autopilot came to life (that’s how we got the flashlogs).
M.

There’s no significant change to Vcc (except when it and everything else disappears), although it is lower than preferred. Typically you should have about 5.1 to 5.3 volts.
And no change to “Flags” which would indicate a fail of one power module and switch over to using the other.
So it’s like a total power loss to the custom carrier board.

I just mentioned the DHSOT thing for the sake of completeness. It’s something you can check out and decide on later. My view is there’s no advantage to using DSHOT1200 so why stress out rates…

I saw the battery voltages were good. You still should set those failsafe values and actions. If the pilot gets busy you cant manually monitor battery voltages (or if you lose the link or something unexpected happens) - you need the copter to be able to look after itself in that regard.

Thank you Shawn. I just looked and we have the thresholds in the parameters (as far as I can tell), but our thresholds are much lower than what you suggest. I’ll talk with our vehicle engineer to make sure we’re not pushing it too far. However, as you pointed out, I think that our actions upon reaching the thresholds have not been set: we will do that before flying next: I agree with you that they are important.

I’m not sure what DHSOT is, but I’ll read on it. I went to field this afternoon with another (identical) frame, and it seems that the vibrations are caused by a combination of the folding propeller and legs: if I swap out any one of them, things calm down. I will swap out the propellers and cross my fingers: I come up with zero alternate scenarios.

The only other thing I suspect (since we changed it) is the herelink2 firmware. However, I was informed that it’s basically impossible to go back in the firmware version. The only reason I suspect that is because I read about the ArduCopter watchdog (introduced after version 4.0.0) which clearly suggests that an “peripheral” can block the main thread (and I believe that the herelink2 remote unit qualifies as a “peripheral”). I have no indication that this is the problem though.

Thanks for the suggestions,
Mihai

The old PWM way of commanding the ESCs is susceptible to noise and timing issues.

DSHOT is a form of digital serial communication with the ESC - essentially commanding some exact percentage of output. It’s noise-immune and handles missing data packets and all sorts of things.
This is the preferred way for a flight controller to work with ESCs now. There is also some other good methods like CAN and onewire, but that’s another story…
The DSHOT rate (DHSHOT300 or 600 and so on) is equivalent to the baud rate in serial terms. So if you want to send commands to the ESCs at a very high rate (like DSHOT1200) then the flight controller needs spare time in it’s loops to do so, and even that is pointless if the the EKF and motor mixer are not requiring any different output from the motors (so DSHOT1200 could just be sending the same motor output commands over and over and over…)
DSHOT600 is even overkill, but it seems to be compatible with more ESCs, and it seems to slip in nicely with the arducopter loop rates.

If you can use Bi-Directional DSHOT the ESC can send RPM data back to the flight controller without any other wiring.
Full DSHOT telemetry data can be accessed by hooking up the ESC “Telem” wire to a serial port Rx pin and some appropriate settings. You get RPM, voltage, temperature and some even do current.
If you’ve got multiple ESCs all their telem wires can be joined into one.
You can also run bi-directional DSHOT and the full telemetry all at the same time.

EDIT:
See this as an example of the telemetry data and it’s importance

If some peripheral locked up the flight controller there should be watchdog info on the SD Card.
See what you can find that via mavftp and grab a copy of everything. Upload and we can look.
Someone like Tridge would be able to confirm if there was a serious issue if we need to call in other people.
In the next boot up messages you would have had watchdog-related messages.

I agree sounds like a prop issue.

Well, I got a second quad falling out of the sky - they share the same hardware, but I’m not sure it’s the same cause. At least the second one did not vibrate, so that’s not it. Also, the second one did fly many tens of missions (possibly hundreds) before turned into a pile of rubble.

Here is the link to the second crash: Quad falls of the sky (a second one)

Shawn, I looked at dshot, and it’s a cool technology. I’m not sure it’s involved directly or indirectly in either of the incidents. BTW, I do have telemetry from the ESCs (in both of the quads that fell), but did not help to elucidate what went wrong - the values from the telemetry show under ESC in the dataflash logs.

Quadzilla, the propellers are of … exquisite quality :-). I use the 17x4 props from T-Motor, here: MS1704-2PCS/PAIR_Polymer Straight_Polymer_Propellers_Multirotor_T-MOTOR Store-Official Store for T-motor drone motor,ESC,Propeller
For the first quad that fell (this thread), all came down intact (it fell from 7m), so I’m ruling them out. For the second one that fell, they all broke, but that thing fell from 50m, so I can’t fault them for breaking.

Check all wires for a short circuit due to abrasion of the insulation. You may have low-level vibrations, but you can’t eliminate them completely. A year ago, my engine burned out while flying in my hex, it was caused by the abrasion of the heat shrink at the gold 3.5 mm contact. Try to make as few connections with contacts as possible and as many solder connections as possible. Place all cables in the armouring. Before assembling the new frame, degrease all screws by dipping them in isopropyl alcohol, then use threadlocker for each screw. The screws are very fond of unscrewing themselves during the flight.

Good advice all around. However, this one was brand new, so it didn’t get a chance to abrade :-). Something else is off with this one. Mark is on it though. The treadlock and oily screws is certainly something to look after: we saw this problem with our screws!

Thanks,
Mihai

Would love to know that your problem is solved, but I still have some minor contribution.

  1. INS_ACCEL_FILTER was set to 20. It is recommended at 10 instead.
    I am not sure whether it led to the incredibly high vibe or not but let’s give it a try next time

  1. You might notice that the vibe on the 1st and 2nd IMU are higer than the 3rd one. This should be another issue.