ArduPilot 4.6.3 - Crash Root-Cause Analysis

ArduCopter Copter 4.6
1

Hello everyone, we had a crash after a long time before our customer tests. We made a root-cause analysis but could not find why. We have seen the SP is gradually increasing, this a potential reason, but would like to hear an expert’s opinion if possible. Normally our system uses exnav, but GPS is also connected. In that flight we wanted to fly with GPS, source was set to SRC2 resp. the GPS. We are using official ArduPilot 4.6.3 Log link: 2026_CrashLog - Google Drive

The case is explained below:

image
image1600×979 211 KB

Our Pilots report:

Based on both the flight log and the attached image, the abnormal behavior started at the marked location. From that point onward, I experienced a partial loss of control. Despite commanding full backward pitch, the drone continued moving forward along the green path. Switching flight modes (AltHold → Loiter → AltHold → LAND) did not stop the forward drift. Eventually, the drone collided with the building and crashed. I would appreciate any insight into what may have caused the vehicle to ignore the commanded input and continue drifting in a fixed direction.

For the interested, visit aridrones.com

2

Was it windy that day?

My observation is that the machine actually tried to follow the commands right until the Land mode was entered. Then, it still tried to do that, but it was too difficult because it also tried hard to keep the GPS position while still descending.

The key obstacle, which in my opinion prevented the machine from being controlled, was an unusually low ANGLE_MAX: you configured the machine to take pitch or roll to be at most 10 degrees. Which is dangerously small if you ask me, precisely because there is too little room for anything, e.g. position holding when battling the wind.

While flying in AltHold, control in pitch and roll was actually pretty reasonable: everything commanded was executed - not without some sloppiness, but quite readily:

newplot-pitch
newplot-pitch1915×1378 312 KB

newplot-roll
newplot-roll1915×1378 265 KB

But whenever the machine entered a GPS-managed mode (Loiter or Land), it just could not keep the position, because the required angle was too small to counteract the external disturbance (e.g. the wind). Everything is in conflict. When in Land, additionally, the authority is further reduced because the machine needs to land, which leaves less room for control.

Solution? Get ANGLE_MAX to its defaults. Pilots need to be trained to fly with this, as well as to land at least in AltHold.

3

The main issue is being overly creative when trying to make ArduPilot safe.

So instead of following community approved, proven safe, procedures like the wiki or AMC, some users take shortcuts and create their own, incorrect tuning and operating procedures.

This is a good example of that, and a good example of the consequences. Do not limit ANGLE_MAX to 10 degrees.

Please download and use ArduPilot methodic configurator software instead.

4

Thank you for your “The main issue is being overly creative when trying to make ArduPilot safe.” %100 subjective review. It helps a lot.

From where and which facts did you assume that we have been creative? What do you think that we have done with our drone, to assume that we assigned some random parameters and destabilized the drone?

5

First of all, thank you very much for your valuable feedback and for taking the time to analyze our logs.

I was the pilot during this flight. Based on what I experienced, I do not believe the incident was caused by the wind. There was no noticeable wind during the flight.

The vehicle started drifting in one direction as if it was intentionally trying to fly there. I continuously applied roll and pitch inputs in the opposite direction of the drift. These inputs slowed the drift down to some extent, but they never stopped it completely.

What was even more noticeable was that as soon as I released the roll and pitch inputs, the vehicle immediately accelerated in the same drifting direction again. Because of this, it did not feel like a normal wind drift. Instead, it felt as though the flight controller was continuously commanding the vehicle to move in that direction.

We intentionally reduced the ANGLE_MAX value because this is an industrial drone. We do not require aggressive maneuvers; our priority is slow, smooth, and controlled movement.

For the same reason, we normally operate in Loiter mode using External Navigation, and we have reduced LOIT_SPEED from the default 1250 cm/s to 120 cm/s. With these settings, the vehicle moves much more slowly and smoothly in Loiter while still being limited to a maximum tilt angle of 10°.

The same 10° ANGLE_MAX has always been sufficient for manual control in AltHold during our normal operations. We have completed many flights with these settings and have never experienced any loss of control or similar behavior before this incident.

However, in this case, the abnormal drift started while the vehicle was in Loiter mode and continued even after switching back to AltHold.

For this flight, the EKF position source was set to SRC2 (GPS). As far as I understand, AltHold does not actively maintain horizontal position. Therefore, we find it difficult to explain why the vehicle continued drifting in the same direction after leaving Loiter based solely on the reduced ANGLE_MAX value.

At this point, we believe that an issue related to the IMU, compass, EKF, or an inconsistency between these systems is a more likely explanation. However, we have not yet been able to identify the exact root cause.

Of course, if we are overlooking something, we would sincerely appreciate your opinion. Thank you again for taking the time to review our case and share your insights.

6

I edited my reply above to be more explicit on what I think went wrong here. There is no way you can evaluate the wind-speed at flight altitude while being on the ground.

Check your RC controller trims. Are they centered?

7

You can verify this in the log. Before switching to Loiter mode, it is possible to see the direction in which the wind was pushing the vehicle. As you know, AltHold does not maintain horizontal position, so it is normal for the vehicle to drift slightly with the wind, and I do not consider that abnormal.

The behavior I experienced started when I switched to Loiter to verify GPS position holding. At that point, the vehicle began drifting, but not in the direction of the wind. I then switched back to AltHold to stop the movement, yet the vehicle continued drifting in exactly the same direction.

We have been developing this platform for approximately three years, and we have never experienced this kind of behavior before.

The RC radio was properly calibrated, including stick centering. We have performed many field tests with this platform. We have also tested Loiter extensively in strong wind conditions. In those cases, the vehicle always fought against the wind as expected, but it never behaved as though it was intentionally trying to fly in one constant direction like it did during this incident.

If you are interested, you are welcome to take a look at our field test videos. You will see that the platform remains very stable even in strong wind conditions.

There is one additional detail that may or may not be relevant. During the accident flight, yaw was provided by the compass while using SRC2, whereas SRC1 uses the yaw estimate from External Navigation. At the moment, these two yaw sources do not match perfectly, and we have not yet found a way to align them completely.

Although the EKF Position Source remained on SRC2 throughout the entire flight, I keep wondering whether the system could somehow have been influenced by SRC1 as well. If that were possible, perhaps it could have affected the compass-based yaw estimate and caused the vehicle to continuously command motion in one direction.

This is only a hypothesis on my part, and I do not have evidence to support it. I simply wanted to mention it because it is one of the possible explanations that came to mind while reviewing the logs and trying to understand what happened.

8

I did not assume that. I explicitly mentioned the one parameter that I think started it.
And I explicitly mentioned that RC-trims, if changed during flight, cause that drift symptom.

Now, with new information from @calikismaill, there are other root possibilities.
Does the externalNAV position drift?

9

When using External Navigation only (SRC1), we have not observed any position drift. In Loiter mode, the vehicle holds its position with centimeter-level accuracy, and we have performed many successful flights with this configuration.

The incident occurred while we were intentionally flying with SRC2 (GPS) to evaluate GPS-based navigation. That is why I am trying to understand whether there could be any interaction between the two navigation sources, even though the EKF Position Source remained on SRC2 throughout the flight.

10

A GPS glich would explain the drift in LOITER, but not the continuation of the drift in ALT_HOLD, so something else is at play here.

I do not have time to analyze your log file in detail. I spend my time improving AMC, but other users can help you out.

11

I agree, and that is exactly what has been confusing us.

I switched back to AltHold specifically to eliminate GPS position control from the equation, but the vehicle continued drifting in the same direction. That is why we do not believe this was caused by GPS alone.

Is there any log message or EKF data that you would recommend checking to determine what could have continued influencing the vehicle after switching back to AltHold?

12

Sorry about the crash. During customer testing is the worst time for it.

Before chasing SP, I’d separate “control problem” from “estimation problem,” because your symptoms smell like the latter. Pull up ATT.DesPitch vs ATT.Pitch for the drift window. If actual pitch is tracking desired pitch (meaning the copter really was pitched back) but it still translated forward, the airframe was doing what it was told. That means the velocity/position estimate it was fighting against was wrong, which points straight at the ExNav to GPS switch. Check XKF innovations and variances, GPS speed accuracy (GPA.SAcc), and whether the EKF was still blending stale ExNav data at takeoff. A bad velocity estimate makes Loiter actively fly the drone “back to where it thinks it should be,” which looks exactly like uncommanded drift that mode changes don’t fix.

If DesPitch and Pitch diverge instead, then look at motor outputs (RCOU) for a saturated or failing motor/ESC.

If it helps, I run these logs through RotorLab (rotorlab.app). The log analyzer is free and it flags EKF innovation spikes and control-tracking divergence automatically. Happy to look at your .bin either way if you post it.

13

Hi İsmail,

I followed the discussion about the position drift during your SRC2 GPS evaluation. I’m developing a local DataFlash report generator called BlackBox, and I ran it against the .BIN log shared in the thread. I thought it might offer a compact, independent view of the flight.

I’ve attached the BlackBox report. It identifies the log as ArduCopter, shows the ALT_HOLD → LOITER → ALT_HOLD → LAND sequence, and flags the later EKF lane-switch/variance messages and crash disarm. It also includes attitude desired-versus-actual, actuator-output, and raw EKF diagnostic plots.

This is not a root-cause diagnosis, and I do not want to overstate what it can conclude. In particular, the report alone cannot decide between an estimator/navigation-source issue and a control or actuator issue. It is intended as a useful companion to Mission Planner/UAV LogViewer and the vehicle’s source/configuration details.

If you have time to review it, I would genuinely value technical feedback: does the timeline align with what happened, and which signals or event windows would make BlackBox more useful for this kind of external-navigation/GPS investigation?

ismail_calik_external_navigation_src2_gps_evaluation_email.pdf (666.7 KB)

Best,
Richard Harris Podolsky
BlackBox

14

First of all, thank you for your interest and for the detailed suggestions. I will also look into the points you mentioned in parallel.

As already shared by @Adil_Evren_Gunay in the original post, the log from this flight is available here:

Please feel free to review the logs yourself as well.

I will also look into the points you mentioned, particularly ATT.DesPitch / ATT.Pitch, XKF innovations and variances, GPA.SAcc, and whether the EKF continued to use any ExNav data.

Also, thank you for recommending RotorLab. Based on your suggestion, I analyzed the log using RotorLab, and I am attaching the analysis report here as well.

One important point that caught my attention during the RotorLab analysis was the occurrence of an EKF_YAW_RESET event during the flight. This makes me think that there may have been a problem or a sudden change in the vehicle’s yaw estimation. I think this could potentially have caused the vehicle to lose an accurate estimate of its heading and consequently drift in a particular direction.

Based on this finding from the RotorLab analysis, I also suspect that the compass calibration may not have been sufficiently accurate. However, I cannot say yet that this was the definite root cause, so I am trying to investigate and verify this point in more detail from the log.

I will also investigate the ExNav - GPS transition, the velocity/position estimate, and the EKF innovations that you suggested. If you notice anything different or particularly relevant when reviewing the log, we would greatly appreciate your feedback.

Thank you again.

crash-report.pdf (24.9 KB)

15

image
image1896×944 175 KB

1. Position innovations (XKF3[0].IPN, XKF3[0].IPE) grow steadily from ~14:58:45 onward

Innovations climb from near zero to several metres before the event. With EK3_SRC2_POSXY=GPS, this suggests the EKF was increasingly rejecting or struggling to fuse GPS position — i.e. less trust in GPS as the flight progressed. XKF4.SP (position variance) rises over the same period.

2. Clear yaw divergence between EKF cores (XKF1[0].Yaw vs XKF1[1].Yaw)

Both cores use the same source set (XKFS.SS=1, same compass MI=0), yet yaw separates by ~50–60° from ~14:58:45. Roll/pitch stay close (~1–2°), so this looks like independent mag/yaw state evolution per core, not a simple attitude difference.

Mag-related messages in the same flight:

  • EKF3 IMU0/1 MAG0 ground mag anomaly, yaw re-aligned @ ~14:58:45
  • EKF3 IMU0 in-flight yaw alignment complete @ ~14:58:51
  • EKF3 IMU1 in-flight yaw alignment complete @ ~14:59:03 (~12 s later)

Both IMUs eventually aligned, but at different times — consistent with XKF2 earth mag field states (MN/ME/MD) diverging strongly between cores while sharing one compass.

working theory: local magnetic interference after the long standstill caused compass/mag field problems. That led to:

  • independent per-core yaw estimates drifting apart
  • degraded GPS fusion (wrong yaw frame → growing position innovations)
  • loss of horizontal control / drift despite pilot input
16

The root cause was clearly identified above. All of this LLM spitballing is just that. Please avoid copying and pasting that sort of thing here, and do not abuse our community by asking for unsolicited feedback on your diagnostic tools.

17

Perhaps it’s time to make a policy change like Stack Overflow has done.
All use of generative AI (e.g., ChatGPT and other LLMs) is banned when posting content.
This includes “asking” the question to an AI generator then copy-pasting its output as well as using an AI generator to “reword” your answers.