Finding cause of crash using only tlog

ArduCopter Copter 4.6
1

Hello,

I would like to request help in finding the most likely cause of a crash incident we experienced, so we ( and maybe others ) can prevent this in the future.

Bit of context:

The copter that crashed is a standard ‘X’ configuration drone, 12.5kg takeoff mass, 32” x 10.6 props, T-motor U8 Lite 85 Kv motors, APD esc’s & PDB, TBS H7 Lucid flight controller running arducopter 4.6.3 stable. The system is built for long endurance missions over water, basic tuning is not finished but the system flies well using the basic tuning structure of the 7" long range build example.

Mission/Flight plan was a simple endurance test, flying a semi-square pattern at 100ft altitude in 15-20kts wind, 26degC outside temp, 10-15kts ground speed. Battery 4P12S LiPo starting at 50V (full). We were planning to fly the pattern from 4.2V/cell to 3.0V/cell, and expected something in the order of 3 hours of flight time in this mass condition and wind.

Flight went as planned, until 28minutes in. Copter was seen to fall away over left-rear side at 100ft altitude, no alarms noticed beforehand, hit open grass field and caught fire. Crew managed to extinguish the fire, but no logging on the copter survived. So the only info available for troubleshooting on what happened is the mission planner tlog, and observations of the crash. This is what I would like to request help on.

Our analysis of the tlog so far…

We looked at requested vs actual attitude, and can see that roll is lost first ( positive direction ), then pitch ( negative direction ). This is consistent with the observation from the pilots.

Then looked at chxout, shows motor 1 ( ps/left rear ) signal going to 1950ms and motor 4 ( sb/right front) going to 1150. This looks to us like a problem with motor 1, that does not follow commanded signal, and again is consistent with attitude readings and observations.

After this, the steps become a bit less certain.

When we look at esc1_current we would expect this to either rise steeply ( in an attempt to follow a 1950ms command ) or go to 0 ( motor wire / signal wire / esc issue ), but the current reading stays more or less the same… which is odd.

Looking at esc1_rpm we would also expect this to either rise steeply or drop off, but this also almost stays the same… and after roll is falling off the reading goes to 4000, then 20000 rpm. The motors are not powerful enough to spin the props at 4000 on the test bench, so these readings are also odd.

Conclusion so far…

Our best guess at the moment is a possible wiring issue with wire carrying the the bi-dir dshot signal from the fc through the pdb to the esc, and that the current, rpm and other info were still coming back through telemetry… esc’s are set to shut down after 500ms of no-signal… but this is not consistent with current & rpm readings staying almost constant.

This also feels like a long shot. Installation was carefully done, as we are well aware this is flight critical, and we have not seen a wire issue after good installation in several years of flying drones.

So, we would like to ask people with more experience in analyzing logs to see if we missed anything?

All help/suggestions are very welcome!

.tlog of the file included as (link to wetransfer tlog)

Boris

PS, if more info is needed pls let me know.

2

No, you did not miss anything:

A wire got loose or,
ESC failed or
Motor failed or,
propeller came off.

There is no way to be sure which of the four caused it.

Are all props intact?
Do the motors still work?

If yes, only the first two options remain.

Doing endurance tests on a not fully configured vehicle has caused similar crashes in the past.
Search this forum, you will find similar reports.

Think about it, you are testing an untuned vehicle for a long time… that is not a good idea.
You want to first tune, then endurance test it. Not the other way around!

3

Hello Amilcarlucas,

Thanks for quick reply! quick response…

Props and motors were almost certainly fine, some fire damage to both so cannot check for sure. Esc’s kept reporting info until well after losing attitude control so something kept working there as well. It is the ambiguity between possible wire failure ( which is unlikely in our experience ) or maybe something else we don’t know yet and that might be in the logs that we wanted to ask advice about.

About the tuning part, I agree that it’s not wise to endurance test a not well tuned machine. The ‘tuning not finished’ statement was referring to our work on long endurance flights, for which tuning can be a significant factor. To test this we use the forum, wiki, 7" long range blog, out own experience and the info from the methodic configurator… and do endurance test flights.

Again thanks for checking,

Boris

4

Motor 3 lost thrust and motor 2, opposite arm, dropped to compensate and down it goes.
Based on this, the servo output assignments and the Frame Type (18)

RCout
RCout2761×1765 192 KB

Those Rate PID’s don’t look right for 2 reasons and I doubt the Notch Filter Configuration is the best choice.

5

@Boris some of your statements I can’t verify from the log, maybe it is my fault.
But I can’t see any ESC data.
Also I don’t understand your answer that motors and / or propellers are worked well up to the end if they are partly destroyed by the fire.
Direct before the power lost off one motor the vibrations increase above 30 on all three axis. All the time vibrations (mainly y and z) rises sometimes above 30 but always only for a single axis at a time. But at that moment right before the thrustlost all three axis rises above 30 upto 45. Right in the moment of thrustlost also the total power consumption goes down.
As I notice first some vibration rises - next controller tries to stabilize by maxout servo1_out and reduce servo4_out - followed by reduced over all power consumption.

6

There is ESC Telemetry Data.

7

I would advise not using the pdb signal traces and run the esc wires directly I had a integrated pdb that had issues with escs randomly stopping mid flight.

8

An APD PDB is solid, I’d trust it’s construction.

My guess is props or motor, since APD ESCs can do far in excess of their rated current.

9

it’s an extra 2 solder joints to save around 20mm of wire. You still need to solder a wire from the pdb to the flight controller so your better to just go direct to the flight controller, and skip the pdb signal routing it’s less to go wrong.

10

@dkemxr you are right, I now saw the individual ESC data on the STATUS Tab while replaying the tlog. Also this shows ESC1 goes up with RPM and going down with CURRENT values.

11

It starts to paint the picture of a broken prop.

12

If the ESC_RPM values are the true (measured) RPMs and not a commanded / calculated /expected value. (I don’t have an experience with such modern ESCs :grinning_face:) than yes, a broken or loose propeller mount

13

I think you guys are right. eRPM is actual RPM from commutation signals so how else to explain a runaway motor RPM, a Commanded high servo output and current dropping to near zero?

Prop failure
Prop failure3086×1862 193 KB

Motor 3 (servo output 1) is the left rear motor for a BF Reversed Frame Type so this jives with the visual report from the pilot.

14

Hello Juergen,

Our idea that the props and motors were, very likely, working well comes from fairly normal current readings ( not extremely high, not 0 ). No mechanical issues, all bolts etc were ok also after the crash. The props even had good lead-lag motion ( they were folding Mejzlik props ).

The high vibrations just before is a good thing to look at, thanks for pointing that out. The drone had just passed a waypoint and was turning ‘into the wind’. In the previous loops that caused it to draw a bit more current at this stage and apparently also generate more vibrations. But valid point to look at!

I dont know where the reduction of power comes from…

Thanks for the help!

15

Hi Shawn, This setup was using an APD500 (500A), and MicroDrive LPi ESC’s (120A) so should be safe from a max current point of view.

16

Hello Boris,
you can clearly see on the ESC data that ESC1 rpm goes up to around 19.000 while the current is nearly 0. This is not saying motor / propeller is working well at that point. The later fire can be issued by ESC2 with 0 rpm while high steady current arround 50A. I think this happend by the crash and blocking of the motor

17

Hi Geo,

We did consider running the wires directly, but it is quite clean to do this trough the pdb. And we were running tlm and 5V / gnd from the pdb to the esc’s making the extra soldering not a very big issue.

For the next build we are thinking of a parallel can wire, that way there is some form of redundancy. We are unsure if that will work with notch filters though so will need to test that setup also.

Thanks for feedback :slight_smile:

18

Hello Juergen,

I am very confident the motors cannot spin the props to 19000 or even the 4000 the trace shows a bit before, at least not in a normal flight condition.

We tested the props and motors on a testbench to 27…2800 rpm after which the motors become hot (60degC). So 19000 must be a wrong reading or the result of very non-normal flight ( which was happening ).

I agree esc2 current is too high, and possibly unsustainable for the motors. The fire started at the lipo’s though, and took out the central bit of the drone including fc and esc’… which is a shame, that would have yielded much more info on troubleshooting this.

Thanks for help!

19

Hi Dave,

I know for sure all props and all motors were still connected to the drone after the crash, props even felt ok in lead/lag.

The motors were damaged by fire, it was still possible to turn them by hand, but not smoothly so it is not possible to exclude e.g. a bearing failure… but that would not yield the graph above I think.

I asked APD/Hargrave to have a look at the logs as well, they are very helpful as always but nothing definitive. Sadly the esc’s were also lost so their logging is also unavailable.

One thing that maybe relevant, the esc was set to shut off 500ms after ‘no-signal’, so that might explain the current dropping away.

One of our theories anyway.

Thanks for help!

20

Yes, perhaps it could. I don’t know if we can tease any more data out of the Tlog and because of the low sample rate it’s hard to know if the timing between data fields is right. There is also a question about the scaling on the graphs. Some are clearly right where other are questionable. Always the issues with Tlogs unfortunately. For example here we see the command output for Motor 3 going to Max and the opposite arm going low which is a typical signature of thrust loss on a motor. It also shows some ms delay before the current drops on that motor but I don’t know if we can really read anything into that.

Current
Current3100×1845 139 KB

One bit of info that is interesting in a gallows humor kind of way is I think you can see the IMU temperature rising as the flight controller burns before logging halts.