Automatic log inspection

[john_skaler]

Hi everyone,

I’m working as a drone software developer for the last 4 years (develop mostly for ardupilot based drones). I found the most difficult part of the job in building, testing, and deploying a drone is analyzing its logs.
Usually, at least in the companies I worked for, a team of engineers sitting and investigating each test flight or flight in which the operator reported a problem or weird events. The investigation is usually done manually and takes a lot of effort and time. The efforts are to be expected when being in R&D state, but the problem is even larger in production when most of the logs aren’t analyzed, and when the problem is noticed it’s usually too late(crash ) .

I was wondering if someone else has encountered the same difficulties and what are the ways you tried to solve them?
Do you know of any automatic log processing tools that give nontrivial insights? I consider Mission Planner visualization and MatLab libraries (like ardupilog) still a manual job, and found some online automatic log analysis but they are not supporting ardupilot.

Thanks in advance for any replies.
john

[rickyg32 (RickyG)]

I confess I have the same concern. I have to rely on the community for assistance and I really would like to be able to run it through an analyzer. There use to be an awesome tool called Log Analizer. Ya I know about the spelling. It was a great tool but it stopped being maintained.
I have not seen anything since.

[Dimitris_Stefanakis]

@Georacer could you jump in and give us a brief about your tools or suggestions for the matter?

[Georacer (George Zogopoulos)]

Hello everyone!

@john_skaler I agree that both Mission Planner and ardupilog are manual tools.
You don’t even know how much I feel for you, and honestly, this is a very hard problem.

There are 3 main cases that I think outline the discussion:

1. A solution for all UAVs
2. A solution for a single autopilot stack
3. A solution for a single production UAV

One tool to rule them all

So, yeah, ideally we would all want to have a tool where we feed our log and it spits out “you forgot to tune your TECS speedweight and your right aileron is about to go loose”.
But we must remember that computers are stupid and at best they can do exactly what we told them to do, in other words they can be as smart as the smartest of us.

So, here’s where I like to use the analogy of the car. Say someone brings you a log of their car ride and they ask you, the human expert, to take a look at it and say how well it performs, without having ever driven the car, or even seen it for that matter.
How are you supposed to know if this particular log has something fishy in it, if this car was mistreated, how well can it perform in its best day, or even if it has gas or cooking oil in the tank!

And even if you try to mix in some model-based smarts, yes, you could expand your diagnostic capabilities, but are you really going to be able to feed it data about propeller specs, aerodynamic performance, inertias, motor configurations and geometries? This kind of data is very costly (in time or money) to come by.

Clearly the complexity is too big here. I wouldn’t even dream of having an automated tool for any random UAV.

This is where the expert’s manual approach can work, though. Much like a doctor, a good inspector can ask the right questions about the system and make the connect the right dots to get some good insights. 100% not repeatable.

One autopilot to rule them all

But how about deep learning and neural networks? If we could pool a lot of flight data and train a model of a “good flight”, then couldn’t an algorithm discern the bad flights?

Perhaps, although the hurdles are still great:

1. You have to pool a lot of data and label them. And despite their reach I don’t see neither Ardupilot nor PX4 pulling this off. The hardest part here is to vet your data, i.e. trust that your data submitters will provide correctly labelled data.
2. You have to have as many labels as the things you look out for. If your only labels are “good flight”, “bad flight”, that’s all you’ll be able to get out of your system. Which is a good start, granted.
3. Data from multicopters likely wont’ be usable for fixed-wing and vice versa.
4. If you want to check control algorithms, there’s a good chance data from past releases won’t be helpful when pooled with data from new ones.

In sum, there’s still too much complexity and manual labor involved.

The inspection tool we deserve, not the one we need

So, sadly, what is left is automatic diagnosis for a specific UAV, whose configuration you can control, for which you can have a lot of well-defined flight data or a good model and for which you could even build a good benchmark performance.

If you do that, and with the right math, you could even have single thresholds checking diagnostic signals. And this goes full circle back to the previous cases where you have a chance of obtaining an exact mathematical model of your system or produce enough reliable data to use machine learning.
But this mostly works! This is what most automatic, online diagnostic systems apply.

The bad news is that by now now we de facto can’t have a diagnostic tool for the whole community. It’s specific to the platform.

My 2c

The actual topic of my PhD was automatic diagnostic procedures in fixed-wing UAVs and what I’d tried was to invent an automated residual generation procedure, which could be applied on UAV models of great detail.
The greater the detail, the better the diagnostic depth.
But I also tried to take a jab at a more general approach: the result was this ICRA paper, where I made a model out of the MAVLink telemetry and tried to assist the human inspector by guiding his eye.
I’d call it medium success, a lot of false positives here.

To do better, I had to have a very good model of the UAV (in the order of 100s of equations) and accurate enough coefficients to build reliable diagnostic signals.
And that’s just for the vehicle. Diagnosing the controller itself is another story for which I don’t have enough experience.

I also tried once or twice to bootstrap a log analyzer project, but it never left the drawing board.

Placebos

So, in the meantime, what can we do to make our lives easier?

One great tool I’ve been using these days that I work with PX4 is PlotJuggler. It is amazing at plotting many simultaneous graphs and give you quick operations and Lua scripting to manipulate the data at a first level.

What’s best about it is that it has plugin decoders for ROS bag files and PX4 ulg logs, so you just open them in the tool. No conversions needed.

Then you make your own layout file, which gives you enough insight at the things you want to check and then it’s a matter of you training your eye. Here are some example screenshots:

If only someone wrote an Ardupilot .bin log decoder for PlotJuggler like he did for Ardupilog…

Another approach I’ve been using lately is to use the Python module for PX4 log decoding, pyulog, to build a small utility library and collection of Jupyter notebooks which make an alternative of PlotJuggler: less flexibility in the visualization but more reusability, data more manipulation, handling multiple logs at once and more automation.

Right now, I try to write one notebook for every small subsystem that I want to analyze, so it’s not exactly automated, but it’s already a start. And mind you, I have only one UAV model to deal with!

So, yeah, I think I’ve rambled long enough. At least I hope something useful came out of it.

Cheers!

[Georacer (George Zogopoulos)]

By the way, I forgot to mention that Auterion do implement a sort of automatic checking in their Suite, which is their online service for aggregating clients’ uploaded logs.
Each plot is marked with a Pass/Warning/Fail flag and all flags give an overall Pass/Warning/Flag for the whole log.
It falls under the first category of solutions I presented in the previous post and it hasn’t worked that well for me. Too many false positives make you ignore the system altogether.

[Verloop (Anthony)]

Wow. And you hope something useful came out of it? This is one of the most useful replies I’ve ever seen. A+++

[john_skaler]

Hi,

Thanx for a very detailed reply!
I agree with you that it is probably very difficult to create accurate solution for every single UAV. I thought that we can create relatively simple rules(or maybe complicated ones) that can detect large amount of common problems(not all of them) for specific flight controller. Example for those common problems can be: bad sensor, battery problems, motor wear and more.
You clearly have more knowledge and experience
than me in the problem and the different ways to solve it so if you say it’s a very difficult problem, i’ll take your word for it.
I’m wondering if your second option would be possible and maybe combine it with the third option. I’ll explain:
Let’s consider a system that receive large amount of logs labeled “good” or “bad” and cause of failure for the bad logs. I assume you can create a neural network that will know to detect the early signs of failure before it happened and generate alerts. I’m assuming here that it is easier to label (maybe automaticly) cause of failure after it happened and that there are early signs that can be detected.
Another idea that can be used is anomaly detection. You can analyze the log history of specific uav or uav type, study its normal behavior and alert for deviation from normal. The detected deviations can be fed into the previous neural net to boost its accuracy.

I’m considering maybe to start implementing the idea myself but it look like it will need a lot of data.
Anyway, i’ll very like to hear what do you think about the ideas and if they sound feasible?

[mishaturnbull]

Hi all,

I couldn’t really have happened across this thread at a better time It just so happens that I’m working on exactly one of these kind of programs, although unfortunately it is specific to ArduPilot DataFlash logs (being built on DFReader.py).

My goal here was to create something I can use both at work and at home for my drone (when I build it have the budget for it, that is) while keeping as much GPL’d as possible and still obeying NDA restrictions. The solution I came up with was to make a UI & main processor that can parse a log and hand out details to a series of plugins, which is where the actual interesting work is done. For example, the main UI would read the log, parse it into a list of messages, and hand the messages off to a processor that would then scan it, for example, for motor imbalances. Plugins can be written by anyone, anywhere, such as in a proprietary environment and not pushed back to the general public, but still have the benefit of not needing to implement a full DFLog parser.

The code is here: https://github.com/mishaturnbull/TrashBin

The name TrashBin springs from another goal of this program - be able to sanitize logs of proprietary information for posting to this online community. I know Mission Planner has a log anonymizer, but I needed a bit more than just location edited and opted to remove certain packets entirely.

I would love to hear your opinions on this setup and if there’s any plugins that could be useful! Multithreading is… in the roadmap, but a definite possibility for better performance. This program is still in its infancy, but at least functional enough that I felt it was appropriate to push a v1.0.0.

Hope this can help someone! Glad I’m not the only one who thought this could be a good thing to have.