@john_skaler I agree that both Mission Planner and ardupilog are manual tools.
You don’t even know how much I feel for you, and honestly, this is a very hard problem.
There are 3 main cases that I think outline the discussion:
- A solution for all UAVs
- A solution for a single autopilot stack
- A solution for a single production UAV
One tool to rule them all
So, yeah, ideally we would all want to have a tool where we feed our log and it spits out “you forgot to tune your TECS speedweight and your right aileron is about to go loose”.
But we must remember that computers are stupid and at best they can do exactly what we told them to do, in other words they can be as smart as the smartest of us.
So, here’s where I like to use the analogy of the car. Say someone brings you a log of their car ride and they ask you, the human expert, to take a look at it and say how well it performs, without having ever driven the car, or even seen it for that matter.
How are you supposed to know if this particular log has something fishy in it, if this car was mistreated, how well can it perform in its best day, or even if it has gas or cooking oil in the tank!
And even if you try to mix in some model-based smarts, yes, you could expand your diagnostic capabilities, but are you really going to be able to feed it data about propeller specs, aerodynamic performance, inertias, motor configurations and geometries? This kind of data is very costly (in time or money) to come by.
Clearly the complexity is too big here. I wouldn’t even dream of having an automated tool for any random UAV.
This is where the expert’s manual approach can work, though. Much like a doctor, a good inspector can ask the right questions about the system and make the connect the right dots to get some good insights. 100% not repeatable.
One autopilot to rule them all
But how about deep learning and neural networks? If we could pool a lot of flight data and train a model of a “good flight”, then couldn’t an algorithm discern the bad flights?
Perhaps, although the hurdles are still great:
- You have to pool a lot of data and label them. And despite their reach I don’t see neither Ardupilot nor PX4 pulling this off. The hardest part here is to vet your data, i.e. trust that your data submitters will provide correctly labelled data.
- You have to have as many labels as the things you look out for. If your only labels are “good flight”, “bad flight”, that’s all you’ll be able to get out of your system. Which is a good start, granted.
- Data from multicopters likely wont’ be usable for fixed-wing and vice versa.
- If you want to check control algorithms, there’s a good chance data from past releases won’t be helpful when pooled with data from new ones.
In sum, there’s still too much complexity and manual labor involved.
The inspection tool we deserve, not the one we need
So, sadly, what is left is automatic diagnosis for a specific UAV, whose configuration you can control, for which you can have a lot of well-defined flight data or a good model and for which you could even build a good benchmark performance.
If you do that, and with the right math, you could even have single thresholds checking diagnostic signals. And this goes full circle back to the previous cases where you have a chance of obtaining an exact mathematical model of your system or produce enough reliable data to use machine learning.
But this mostly works! This is what most automatic, online diagnostic systems apply.
The bad news is that by now now we de facto can’t have a diagnostic tool for the whole community. It’s specific to the platform.
The actual topic of my PhD was automatic diagnostic procedures in fixed-wing UAVs and what I’d tried was to invent an automated residual generation procedure, which could be applied on UAV models of great detail.
The greater the detail, the better the diagnostic depth.
But I also tried to take a jab at a more general approach: the result was this ICRA paper, where I made a model out of the MAVLink telemetry and tried to assist the human inspector by guiding his eye.
I’d call it medium success, a lot of false positives here.
To do better, I had to have a very good model of the UAV (in the order of 100s of equations) and accurate enough coefficients to build reliable diagnostic signals.
And that’s just for the vehicle. Diagnosing the controller itself is another story for which I don’t have enough experience.
I also tried once or twice to bootstrap a log analyzer project, but it never left the drawing board.
So, in the meantime, what can we do to make our lives easier?
One great tool I’ve been using these days that I work with PX4 is PlotJuggler. It is amazing at plotting many simultaneous graphs and give you quick operations and Lua scripting to manipulate the data at a first level.
What’s best about it is that it has plugin decoders for ROS bag files and PX4 ulg logs, so you just open them in the tool. No conversions needed.
Then you make your own layout file, which gives you enough insight at the things you want to check and then it’s a matter of you training your eye. Here are some example screenshots:
If only someone wrote an Ardupilot .bin log decoder for PlotJuggler like he did for Ardupilog…
Another approach I’ve been using lately is to use the Python module for PX4 log decoding, pyulog, to build a small utility library and collection of Jupyter notebooks which make an alternative of PlotJuggler: less flexibility in the visualization but more reusability, data more manipulation, handling multiple logs at once and more automation.
Right now, I try to write one notebook for every small subsystem that I want to analyze, so it’s not exactly automated, but it’s already a start. And mind you, I have only one UAV model to deal with!
So, yeah, I think I’ve rambled long enough. At least I hope something useful came out of it.