I imagine of a utility software which can act as a database for logs. Some of its features would be:
Database-only functionality: Its core is about storing and searching logs, not processing them.
Local storage: The logs are on your PC and databases are portable by physical means (USB sticks etc).
Log-agnostic: Not specific to ardupilot logs (dataflash or tlogs)
Metadata-based: Since any log file format is admissible, metadata fields contain vital information. Also useful to record external quantities (notes, pilot, platform, events, etc)
Extensible: It provides an API for implementing log-processing plugins.
Do you know if such a thing exists?
I know that a lot of web-based utilities exist but do not fit the requirements and most are closed-soure anyways.
If not, is anyone interested in discussing some specifications?
Hey @Georacer, there are instructions here for running it without the rest of the maverick stuff:
Interested in what kind of thing you had in mind? I’m hoping to sit down in the next month or two and revisit logging. I was looking to do a client-only solution like the plot.dron.ee site (but open source), but there are lots of cool things you can do with a database-driven system like Maverick currently uses that you could never really do client-only.
I’m in the end of a PhD on UAV fault diagnosis and at some point I’d like to ask the community to voluntarily contribute logs for processing and analysis on common crash causes.
In support of this, I imagine a Python module for log management which one could import in order to pull in the log database functionality to your analysis front-end software.
To recap, what I think would be useful would be a (Python) library which would cover all of the features of the first post.
Possible applications would be:
Plain log storage: I have a lot of logs and I need to go search and review them, similar to Dronee and other log book apps.
Log analysis: More involved log examination algorithms, akin to Maverick and Log Analyzer.
One-off research: You put a bunch of logs in one place (regardless of where you found them) and write your tailor-made analysis on the issues you are interested in, such as my case. For example, inferences on mean flight time across UAV classes, which APM version had the most crashes or (in my case) generate diagnostic signals and test their accuracy on real logs.
I imagine this as a vault, where one would put in logs, along with a bit of information, and the module would parse for rudimentary information. In addition to that, it would provide a useful API.
So, in summarily, I imagine this API kind of like:
Parsers:
Total number of logs
Total flight time
Number of different vehicles
etc…
Getters (for any individual log):
Any quantity timeseries, properly timestamped
Metadata, useful for filtering and cross-correlation, as said before (UAV ID, battery ID, pilot, date, etc). This information could be stored in a document-based DB, such as TinyDB.
The whole log, for database migration but with some filters applied (possibly based on the aforementioned metadata, e.g. get me all the logs of this FW version).
I imagine specifying a directory under which you would create one folder for each log and one JSON file alongside it, containing the metadata.
@Georacer v.interesting!
So the Maverick analysis front-end is simply grafana with some custom dashboards, that get written to by a backend python process (mavlogd). Grafana is connected to the backend influxdb database where it gets all of it’s data. influxdb was chosen because it can store non-averaged data down at the nanosecond level.
The main work is parsing the log files and getting the data into influxbd. Once the data is there, you can query it at will with a rich query language as well as through language bindings. You can tag each logfile and then select the data by logfile, vehicle, time etc, which sounds close to what you’re looking for. Here’s the script that translates the log files into influxdb - it uses pymavlink dfreader for the underlying log reading/parsing:
ps - this has been my dilemma for the next step for this project. Tools like plot.dron.ee are cool because there’s essentially no installation needed - it’s all done in the client. However, you’re limited to client-side technology (which is improving by the day) and it’s much more difficult to do large scale analysis across flights/vehicles.
I’ve been trying to think of how to do a tool that will cater to both.
I normally use a github issue or gitter to discuss these kinds of things but they’re not very structured. Happy see whatever you have in your mind
Love your Ardupilog - I could see something similar in a high level python module onto eg. an influxdb backend with a predefined schema. That would be fab.
@Georacer Sorry for lack of response - been very busy and my todo list just seems to get ever longer! Hoping to sit down over the next couple of weeks and look at this closer, and perhaps thrash out a PoC. I do think quite a bit of this has already been done in Maverick, in some form or another.
I’ll definitely take peaks from your Maverick implementation of influxDB and log loading, but I really need a standalone Python library to interface with it, possibly through Pandas frames.
If you really have done everything I need in Maverick, I’ll just port it to a library and be done with it!
hey, can you help me out, I want to plot a graph of waypoint and Acutal path of UAV on the same graph with Latitutes and logitutes as y and x axis. How do I get the data from the mission planner app to plot this type of graph?
