Copter flipped and crashed - EKF3 innovations high - seeking help for analysis

ArduCopter Copter 4.6
1

A couple of days ago I posted a report about a battery range test flight that experienced a weird increase in X-axis vibes on IMU0 and IMU1 (Orange Cube) and a few minutes later, high cpu load. The flight however was uneventful. (IMU 2 did not show an increase to vibes)

Today I decided to repeat the test flight to see if it would reoccur.

This flight did something different - about 14 minutes into the flight, the copter flipped and came down. (on long soft grass - only damaging the landing gear)

I’ve been looking through the BIN file to see if I could understand what happened.

I’m not all this knowledgeable in this area - but it looks to me like there was a EKF3 lane swap do to high innovations, a EKF3 failsafe to LAND mode, and loss of control as the copter fell.

Here are graphs of Lane 1 and 2 innovations at the time of the crash:

lane 1 innovations
lane 1 innovations1920Ă—960 98.2 KB

lane 2 innovations
lane 2 innovations1920Ă—960 120 KB

I’ve uploaded the BIN file here: https://drive.google.com/file/d/111tuIpvOFUXL0wIIUM3ErecjMEexYezk/view?usp=sharing

And for convenience, here’s a text file of the mavilink messages:

MavLink Messages from crash - Battery Range Test until Flight Controller quits - 2025-05-28 16-29-18.txt (24.4 KB)

I haven’t had any response yet to my post about the weird IMU0 and IMU1 vibes and high CPU load from my last test flight - but I got the feeling that maybe it was a hardware issue with the cube.

But now I’ve had my first two flights on Copter 4.6 go wrong - so I’m wondering if there’s an issue now with 4.6 firmware.

I’d appreciate any and all help on the analysis. Thanks!

2

What you noted is after the fact. Thrust loss on Motor 3 in sync with an ESC error. Motor 4 drops and it’s done. Lane switching, EKF errors, Vibe comp and FS action comes after this.

RCout error
RCout error3385Ă—1040 212 KB

1 Like
3

Thank you Dave. Excellent - I haven’t charted the ESC.nn.ERR parameter before. The wiki says very little about it - other than it’s a “rate” no an absolute.

image
image609Ă—463 26.7 KB

If you looked at my last post where I noted a sudden increase in X-axis vibes for IMU0 and IMU1, (no change for IMU2) there was also an increase in CPU load that happened a few minutes later.

The raise in CPU load is associated with ESC errors on this previous flight.

image
image1895Ă—1011 54.3 KB

So that establishes a relationship there.

On today’s flight - it’s different - the CPU load hovers around 400 the entire flight.

image
image1881Ă—1011 51.2 KB

So the question I propose from this: Do the ESC errors come from a failing ESC, or is there something about the Copter 4.6 BD-Shot firmware that causes it.

Maybe the only way to know without swapping out the ESC is to downgrade to the previous version of firmware.

My longest flight on 4.5.7 firmware is only about 6 minutes - and it shows no ESC errors.

image
image1203Ă—566 29.6 KB

I hate to downgrade to an older version of firmware - but maybe that’s the only way to better isolate the problem.

I’d appreciate any suggestions on troubleshooting steps.

4

Yes, agree with that sentiment. You might try a different Dhsot rate (SERVO_DSHOT_RATE). Bump it to double loop rate and see what happens.

5

Good idea. Thanks.

I’m using an AM32 ESC - I’ll also check for a firmware update for the ESC.

6

FYI @andyp1per @bugobliterator

7

@jstroup1986,

I agree with @dkemxr that this appears to be a failure of the front left motor (aka Motor3).

We see the classic signs:

  1. the desired vs actual roll and pitch angles diverge suddenly after having been close together until then
  2. the actual vehicle rolls forward and left towards motor 3
  3. the forward-left motor’s (Motor3) output goes to maximum while the opposing motor goes to minimum

image
image1443Ă—884 53.6 KB

I think this may be the first time I’ve seen a log of a motor failure on a vehicle with ESC feedback. I’m sure others have seen this but it’s a first for me. Some interesting things we see:

  1. the ESC index is one-off from the RCOU field names. So for example ESC9 corresponds to RCOU.C10. This is slightly annoying but I can imagine why it’s happened. We normally use “0” base but for the RC outputs we use “1” based because it matches the labels on the flight controller.
  2. Motor3’s (ESC10’s) RPM output flat lines (see blue line below). This indicates to me the ESC has stopped working completely and is not even providing information anymore.

image
image1443Ă—884 55.7 KB

  1. ESCs 8~10 start reporting errors nearly 4 minutes before the crash. This suggests we should provide more visible warnings to the pilot

image
image1443Ă—884 58.3 KB

Thanks very much for this valuable log!

EDIT: I’ve created issue/to-do request for both Mission Planner and QGC to add improved ESC status reporting.

8

@jstroup1986,

It also looks like the GPS (node 125) is spamming the autopilot (and GCS) with messages. If this is a Here GPS could you try updating it’s firmware as well? I’m pretty sure this was resolved in recent firmwares. You can see all the spam in the little white labels on the screen shots I’ve posted above

9

One last thing is I wonder if the ESCs are extremely hot or we have an issue with temperature reporting for DShot ESCs.

image
image1582Ă—897 55.5 KB

From a recent test on my own vehicle using AM32 ESCs with DroneCAN I get much more sensible temperatures

image
image1204Ă—568 34.7 KB

Anyway, I’ll raise this issue with other devs

10

Extremely likely I would say. Heat in the ESCs or motors causing failure.

1 Like
11

To clarify the temperature reporting graphs I’ve posted above. Apparently AM32 ESCs temperature reporting is the ESC MCU temperature not the mosfet temperature.

@jstroup1986, I wonder if you’d be happy to show how the ESCs are placed and whether they’re wrapped or in a case, etc? No pressure of course, I’m just interested as to how an ESCs temperature could get this high.

12

Joseph had reported this temp anomaly, if that’s what it is, previously and I has suggested perhaps it’s the same issue seen with some other ESC processors (F4’s).

1 Like
13

Yeah F4’s known to be buggy

1 Like
14

@dkemxr, @andyp1per,

Thanks for this feedback. So is the “F4 buggy” issue a hardware issue or is it an AM32 issue? Sounds like it’s hardware which would be sad because that would be harder to fix

15

I believe it’s hardware for the F4 MCU’s as seen in the Tekko F4 Metal units. The Skystars ESC that Joseph is using has a GD32E230 MCU but I don’t know if that’s relevant to this issue or not.

1 Like
16

Its an F4 silicon bug. But I am not aware of any other MCU’s having this issue.

1 Like
17

Thank you @andyp1per @rmackay9 and @dkemxr for the excellent analysis.

It finally stopped raining enough here to do more test flights.

For reference, here’s the quad in question:

teardrop copter photos
teardrop copter photos1920Ă—713 199 KB

The ESC, a T-Motor Tekko32 F4 4in1 Mini 50A, sits in a 15mm space between the frame and the flight controller.

This frame was originally designed for the Holybro Pixhawk 6C Mini flight controller - this is an experiment to see how well it works with an Orange Cube.

The question of ESC temperature has come up before, notably with a Skystars KM55. But other 20x20 ESC’s have been OK. I was aware of the issues of the F4 reporting data. Fortunately, RPM data seems reliable.

I have tried to adopt AM32 ESC’s - but as I’ve noted in other posts, AM32 doesn’t seem to play well with ArduPilot. AM32 was designed for rovers. The advanced features that BLHeli_32 introduced don’t seem to work properly on my quads.

With the ESC errors that @dkemxr identified, I focused on the ESC’s firmware. I checked it ensure it was on the latest release - it is. Then I looked at the settings I’d selected. I decided to de-select the two remaining advanced options. These were my settings:

am32 before
am32 before1337Ă—800 84.8 KB

I deselected Auto Timing Advance - and set 15 degrees. And I deselected PWM Type - By RPM, and kept PWM Frequency at 24 kHz.

am32 after
am32 after1345Ă—805 76.4 KB

My thought was that perhaps by reducing the load on the ESC’s processor, it might generate fewer errors.

Lucky guess - I flew first a 10 minute and then a 20 minute test flight with no ESC errors.

Here’s a plot of the ESC errors (none) and the CPU load:

image
image1873Ă—1012 46.9 KB

Thank you @rmackay9 about the tip regarding updating my Here3 firmware.

One other thing I did notice is that the IMU temp gets quite warm. I don’t know if this is OK - or a thread and should be addressed.

image
image1901Ă—1004 40 KB

Here’s a link to the BIN file for this 20 minute test flight:

I would appreciate any an all comments moving forward - especially regarding AM32 and the IMU temps. Thank you.

18

One more thing -

I reported earlier that X-axis vibes on IMU0 and IMU1 seem to increase - while IMU2 does not.

image
image1852Ă—1009 68 KB

Maybe there’s something about the IMU isolation material on this Orange Cube that’s starting to deteriorate. @CraigElder has suggested just replacing it.

19

Hi @jstroup1986,

The IMU temp of 65 degrees is fine. There’s a BRD_HEAT_TARG parameter that can be used to configure this but 65 is an acceptable value.

I think AM32 ESCs should work fine and it looks like you’ve got it working now. Tridge demonstrated them at the Kaga developers conference and I have an aerial photography copter that uses them (although I haven’t flow it a lot yet). I use AM32 ESCs with DroneCAN though, not the BLHeli communication protocol. I must admit that I don’t know much about the AM32 settings yet.

20

No need to intentionally make the board so hot (by increasing BRD_HEAT_TARG), you just want to ensure that it still has a valid/compensated calibration when it does warm up. Probably best to leave BRD_HEAT_TARG at default, but perform IMU temperature calibration if you plan to fly above that value on a routine basis.

Boards with heated IMUs somewhat alleviate the need to perform a temp cal, but this is a case where it’s prudent to do it.

1 Like