Introducing an AI-Assisted Log Diagnosis Engine: Tracing Root Causes in .BIN files

ArduCopter
1

Hello ArduPilot community & developers,

Over the last few weeks, I have been building a hybrid diagnostic engine designed to automatically analyze ArduPilot .BIN flight logs and help identify the root cause of crashes faster. I’m sharing it here to get your feedback on its accuracy and to see if this approach could be useful for triage.

:rocket: What the Tool Does

Analyzing a .BIN file after a crash takes deep expertise. My goal is to partially automate this by combining ArduPilot’s known safety thresholds (Rule Engine) with a calibrated XGBoost model trained on historical crash telemetry.

Currently, the engine evaluates over 150 features and diagnoses 9 different failure classes (e.g., motor_imbalance, compass_interference, ekf_failure, vibration_high, etc.). It outputs a confidence percentage along with the exact numerical telemetry evidence that triggered the diagnosis.

:magnifying_glass_tilted_left: Proving it on 34 Unlabeled Forum Logs

To test how well the tool works on wild data, I took 34 crash logs scattered across ArduPilot Discuss (logs where the thread title simply said “Oscillation Crash”, “Flyaway”, or “Failsafe”) and ran them completely blind through the hybrid engine.

The most interesting finding: The symptom in the thread title is rarely the physical root cause. (e.g. 7 out of 8 “oscillation_crash” logs were diagnosed as compass or motor issues).

I have numbered the 34 engine results below, along with the links to the original posts.

:speaking_head: Calling all Log Analysis Experts!
Could you please take a close look at these 34 results and tell me if the engine’s diagnosis might be correct? It would be incredibly helpful if you could reply with something like: “For log #14*, the engine is right/wrong because of X.”*

:red_circle: Oscillation Crashes (Engine found Compass or Motor issues)

1. Oscillation In Roll And Pitch Resulting In Crash — log_0048_oscillation_crash

Engine: compass_interference (85%) | Evidence: mag_field_range=433.05

2. Quadcopter Crash Log Analysis Request — log_0049_oscillation_crash

Engine: compass_interference (85%) | Evidence: mag_field_range=526.61

3. Quadcopter Crash Log Analysis Request — log_0050_oscillation_crash

Engine: crash_unknown (0%) | Reason: Log too short, no features triggered.

4. Crash When Switching From Rtl To Loiter — log_0056_oscillation_crash

Engine: compass_interference (85%) | Evidence: mag_field_range=510.73

5. Crash When Switching From Rtl To Loiter — log_0057_oscillation_crash

Engine: compass_interference (83%) | Evidence: Rule + ML agreement.

6. Tuning Issues With Plane — log_0052_oscillation_crash

Engine: compass_interference (55% / Weak) | Evidence: mag_field_range=276.18

7. Tuning Issues With Plane — log_0053_oscillation_crash

Engine: motor_imbalance (81%) | Evidence: motor_spread_max=963.00

8. Tuning Issues With Plane — log_0054_oscillation_crash

Engine: motor_imbalance (81%) | Evidence: motor_spread_max=963.00

9. Tuning Issues With Plane — log_0055_oscillation_crash

Engine: motor_imbalance (81%) | Evidence: motor_spread_max=963.00

:orange_circle: Flyaways (Engine found Motor Imbalance or EKF Failures)

10. Multiple Flyaways In Guided — log_0058_flyaway

Engine: motor_imbalance (47% / Weak) | Evidence: motor_spread_max=810.00

11. Something Is Wrong With My Copter Build — log_0059_flyaway

Engine: motor_imbalance (55% / Weak) | Evidence: motor_spread_max=507.00

12. Something Is Wrong With My Copter Build — log_0060_flyaway

Engine: motor_imbalance (47% / Weak) | Evidence: motor_spread_max=521.00

13. Multiple Flyaways In Loiter And Auto — log_0061_flyaway

Engine: ekf_failure (85%) | Evidence: ekf_vel_var_max=7.13, ekf_pos_var_max=1.69

14. Quad Flyaway Need Analysis — log_0062_flyaway

Engine: motor_imbalance (85%) | Evidence: motor_spread_max=1000.00

15. Possibly A Fly Away Please Take A Look — log_0063_flyaway

Engine: motor_imbalance (93%) | Evidence: motor_spread_max=916.00

16. Rtl Flyaway Bad Pos Among Others — log_0064_flyaway

Engine: compass_interference (85%) | Evidence: mag_field_range=912.49

17. Radio Failsafe During Operation — log_0065_flyaway

Engine: compass_interference (85%) | Evidence: mag_field_range=510.73

18. Radio Failsafe During Operation — log_0066_flyaway

Engine: motor_imbalance (81%) | Evidence: motor_spread_max=950.00

19. Solo Flyaway And Crash During Auto Mission — log_0067_flyaway

Engine: compass_interference (81%) | Evidence: mag_field_range=411.68

:yellow_circle: Failsafes / Miscellaneous

20. Rtl Does Not Return To Launch — log_0028_battery_failsafe

Engine: motor_imbalance (81%) | Evidence: motor_spread_max=838.00

21. Pixhawk Clone Motors Not Running Same Speed — log_0029_battery_failsafe

Engine: gps_quality_poor (85%) | Evidence: gps_hdop_mean=99.99

22. Copter 3.5.5-rc1 Beta Testing — log_0030_battery_failsafe

Engine: rc_failsafe (76%) | Evidence: evt_rc_failsafe_count triggered

23. Quadcopter Gives Radio Failsafe And Climbs — log_0069_radio_failsafe

Engine: compass_interference (58% / Weak) | Evidence: mag_field_range=3136.32

24. Failsafe Sbus Not Working — log_0070_radio_failsafe

Engine: motor_imbalance (81%) | Evidence: motor_spread_max=900.00

25. Failsafe Sbus Not Working — log_0071_radio_failsafe

Engine: gps_quality_poor (85%) | Evidence: gps_hdop_mean=99.99, gps_fix_pct=0.00

26. Failure To Launch — log_0032_hardware_failure

Engine: compass_interference (76%) | Evidence: mag_field_range=422.83

27. Copter Unexpected Rapid Climb After Arming — log_0068_uncontrolled_descent

Engine: compass_interference (85%) | Evidence: mag_field_range alert

:black_circle: Additional Generic Forum Logs

28. Vibration and Notch Filter Help — 00000058.BIN

Engine: compass_interference (85%)

29. Crash at Takeoff Investigation Help — 00000060.BIN

Engine: compass_interference (55% / Weak)

30. Sending Unknown Message 44 — 2016-09-18_08-44.bin

Engine: compass_interference (47% / Weak)

31. Prearm Crashdump Data Detected — 2016-09-18_16-01.bin

Engine: compass_interference (47% / Weak)

32. Getting Firmware Version from BIN File — 2018-09-21_18-26.bin

Engine: motor_imbalance (47% / Weak)

33. A Tail Servo and RC Reversed Settings — 40.BIN

Engine: compass_interference (49% / Weak)

34. flash.bin — Skipped (firmware bootloader artifact, not a flight log)

:globe_showing_europe_africa: Next Steps

Every log this engine uses to learn is rigorously mapped back to its original thread. The engine was already trained on over 100+ other structured flights from this forum, and the 34 you see above are completely new, untrained “wild” logs.

Thank you so much to any experts who have the time to help me verify if the engine is on the right track!

edit:- i dont want you to check again,i want your answer that if inference that i got is good enogh for you to work with or not working like (simply i am on right path or not OR my path is too broken that not even worth moving).

Best regards,
Agastya

1 Like
2

I reviewed the 1st log on your list and if you would have reviewed this log yourself it would be evident that “compass interference” had nothing to do with the cause of the crash. It rarely does so I would not bother looking at other logs where your engine produced that result. This craft was underpowered to start with and by the end of the flight the motors were being commanded to maximum producing the very easy to see Thrust Loss errors. Stability is also lost under this circumstance.

Surly if you are developing this tool you can read the logs manually and determine for yourself what the root cause is?

1 Like
3

I took a peek at the last log (33), and it’s the same incorrect, hallucinated compass interference result. The root cause is identified in the topic from which it originated.

I agree wholeheartedly that this is a requisite for developing such a tool.

1 Like
4

Thank you for taking the time to look at Log 1 and identifying the true root cause (Thrust Loss / underpowered craft). This is exactly the kind of domain expertise and reality-check I was hoping to get by posting here.

To answer your question about why I didn’t manually determine the root causes beforehand: my primary goal with this batch was to run these 34 “wild” logs completely blind through the automated engine and post the raw, unfabricated results exactly as it produced them. I specifically chose not to manually intervene, cherry-pick the logs, or override the engine’s mistakes, because I wanted to present an honest, unfiltered baseline of where the tool currently stands to see its blind spots.

And your review perfectly exposed its biggest blind spot. You are absolutely right—the engine clearly has a systemic bias. It is far too sensitive to compass_interference (likely misinterpreting the magnetic noise during the physical crash sequence itself as the cause of the crash).

Because of your feedback, I can see what I need to fix:

  1. The engine completely missed the Thrust Loss because I haven’t yet built the feature extraction rules to flag when motors are being commanded to maximum output without corresponding stability/lift.
  2. I need to severely dial back the engine’s sensitivity to compass fluctuations.

Posting raw, unfiltered results is intimidating exactly because experienced mentors like you will immediately spot the errors, but I believe it’s the only way to genuinely improve the pipeline without faking its capabilities.

Thank you again for the direct feedback—I will be adding “Thrust Loss” detection to the feature extraction pipeline this week!

5

Even your forum posts appear to be AI/LLM generated. How are you approaching the development of this project? Is it entirely AI generated/vibe coded?

6

I think the only way is to first know what you are looking at by developing the knowledge and experience yourself before attempting to train a model blindly.

And what is “urgent”?

1 Like
7

Thanks for checking those out guys. You are 100% right — the engine completely hallucinated on 1 (Thrust Loss) and #33 (Reversed RC settings).

To answer your question: I do know how to read the logs manually, but for this specific batch, I intentionally ran them completely blind through the engine and posted the raw, unfiltered output. I really didn’t want to step in and fix the engine’s mistakes before posting, because I didn’t want to cherry-pick the data or artificially inflate how well the tool is doing just to make it look good. I wanted to see exactly where it fails when left to its own devices on wild logs.

And you guys found its biggest blind spot immediately. The engine is clearly way too trigger-happy with compass_interference. It looks like it saw the crazy magnetic noise generated during the physical tumbling of the crash itself, and mistakenly assumed that was the cause.

Plus, since I haven’t written rules or feature extractors yet for things like pegged motor throttle (Thrust Loss) or backward RC settings, the engine is just getting confused and defaulting to the compass.

Posting raw results like this is a bit embarrassing when the engine gets it so obviously wrong, but getting blunt feedback from experts like yourselves is the only way I can actually fix these systemic biases in the code. I didn’t want to fake the results.

I really appreciate the reality check! My immediate next step is going to be dialing way back the compass sensitivity, and adding new checks to detect Thrust Loss and basic setup errors. Thanks again.

8

(RESPECTFULLY ) I DONT KNOW HOW TO “structure a post thats why i took help”,and yes you can call it vibe coded as i took help of an ai (decisions were mine like what to use,research was mine like how model would treat (ways to get my answer is correct are not )i just dont want others to copy thats why i didnt post github repo although i gave it to one of discord mod ). He always told me look for hallucinations although my program is that well i just wanted to show i made some thing my solution to your problem can be wrong but intentions arent .

9

You don’t need to structure or format a post in any specific way - just use your own words. Here, the words themselves simply appear to be the direct output of an LLM, quite verbose, somewhat patronizing, and frankly, not particularly helpful overall.

You haven’t done any of the due diligence to even read the forum topics where these issues originate, many of which point to the root causes of the issues presented. This is at best low effort, likely low quality, and it is rather abusive of the community to expect of others the due diligence you have not bothered to execute.

1 Like
10

#21 is also funny. What has in that case the Motor speed to do with GPS.
This AI is only wasting many resources of our world to only produce CO2 and global warming. And finally, you also expect the experts to sacrifice their free time again, after they have already made a significant and meaningful contribution to the actual topics, to correct the errors of a less intelligent AI.
This is not my world

1 Like
11

i dont expect them to do i need help if they can help me/want to help me.if thats the case i am also wasting my time posting what i have achieved although not good enough (i know this before hand)

12

the only thing i understood from your reply is what i made is to bad to even thowing it would be a waste of time , i should just leave it.(i wont i will make it better)

about the i actually understood how the data is provided the logs is divide in 5 parts (sadly yes i know how to read them and my answers doesnt show it i will post my notes here for proof would that be better hand written so that you cant say i made that also by ai)

13

I picked a couple of the logs and found the were logs that weren’t crashes or incidents, but people requesting help with tuning or filters. For example 28, 32, 33. We have tools for that already and this kind of AI slop isn’t going to help.

1 Like
14

I see on the Discord GSoC channel where you got the idea to create a forum post presumably to “engage” which is encouraged. But if you had lurked on the forum for a bit you probably would have discovered why this implementation was never going to be successful. I suggest you abandon this thread and start over perhaps using a log from your own craft and after your tool can actually do something useful.

And I would suggest to not put “urgent” in any post. All volunteers here.

4 Likes
15

Based on my observations of this forum and some others, in some countries/cultures (I would not point a finger to which exactly, but you may guess) it seems that you have to request an urgent reaction in order for someone to do something at all.

Which of course is not an excuse for anything, including the kind of slop enforcement we have here. Rather opposite, if someone asks something to be done urgently, it appears that it’s best to ignore the request altogether, because any help would be interpreted as a sign of weakness, any question as a sign of incompetence, etc. etc. All the cultural things, you know.

16

hey really appriciated that you can help me what exactly problems you want an ai to be able to find, i want it to be like you put the data it can predict if that data(which is given my user to diagnos) dont have to go anywhere else just understand what exact problem is and can see hand book what one should chek )as if my approch is wrong please tell me ?

17

Gemini said

So, the main problem was (urgent): I removed the other part. Can you see the placement where I put that, and why is it not a heading? I put that word in, as I mentioned earlier on Reddit, because you won’t get a reply as fast here if you don’t use that word for the help I might need. That’s how I should ask. All the experts who have seen my project post and replied with good criticism have my genuine respect.

18

Personally, I don’t. I take pride in the skills I’ve developed learning to read log files, identifying problems and helping users to solve them. What you’re doing is somewhat insulting.

There are thousands of variables that can cause trouble or require fixing. Many are not in the parameter/log file, but in the context of the setup, the situation, and the user. What you should do is focus on a specific area of the log files. Perhaps tuning or compass calibration. Learn to solve the problem yourself, find data sets on that specific issue to test with, then create a tool to help you.

And if you think that the volunteers here are going to look over 30+ logs (Many of them we already posted the solutions too…) to help you validate your tool, you might wish to reconsider your approach.

3 Likes
19

Seriously? What does it say for The Rule of Holes? No reply required and I would suggest none.

1 Like
20

let me answer all the those one by one

  1. insulting it shouldnt be i dont want you mods to be asked dumb quetions by morons like me.i am just trying to give you all a helping hand that will be there for anyone if something happens to this discussions forum(its like how you use ai now you even believe in ai more than you school teachers for there answers but teacher answer will be more correct because of that experience they got.i never want it to be insulting in any way man.

  2. why every body is saying i never understood the log files i think missing something i am able to understand logs and i know i how to uderstand with the graphs

my post is just showing how bad my product is and i want experts to have i glance and ask me what i should do to make it better

*check the edited part

atp i think the way i thought was better aproach to do my corrections of my program .*