Hey @Georacer, there are instructions here for running it without the rest of the maverick stuff:
Interested in what kind of thing you had in mind? I’m hoping to sit down in the next month or two and revisit logging. I was looking to do a client-only solution like the plot.dron.ee site (but open source), but there are lots of cool things you can do with a database-driven system like Maverick currently uses that you could never really do client-only.
I’m in the end of a PhD on UAV fault diagnosis and at some point I’d like to ask the community to voluntarily contribute logs for processing and analysis on common crash causes.
In support of this, I imagine a Python module for log management which one could import in order to pull in the log database functionality to your analysis front-end software.
To recap, what I think would be useful would be a (Python) library which would cover all of the features of the first post.
Possible applications would be:
Plain log storage: I have a lot of logs and I need to go search and review them, similar to Dronee and other log book apps.
Log analysis: More involved log examination algorithms, akin to Maverick and Log Analyzer.
One-off research: You put a bunch of logs in one place (regardless of where you found them) and write your tailor-made analysis on the issues you are interested in, such as my case. For example, inferences on mean flight time across UAV classes, which APM version had the most crashes or (in my case) generate diagnostic signals and test their accuracy on real logs.
I imagine this as a vault, where one would put in logs, along with a bit of information, and the module would parse for rudimentary information. In addition to that, it would provide a useful API.
So, in summarily, I imagine this API kind of like:
Total number of logs
Total flight time
Number of different vehicles
Getters (for any individual log):
Any quantity timeseries, properly timestamped
Metadata, useful for filtering and cross-correlation, as said before (UAV ID, battery ID, pilot, date, etc). This information could be stored in a document-based DB, such as TinyDB.
The whole log, for database migration but with some filters applied (possibly based on the aforementioned metadata, e.g. get me all the logs of this FW version).
I imagine specifying a directory under which you would create one folder for each log and one JSON file alongside it, containing the metadata.
So the Maverick analysis front-end is simply grafana with some custom dashboards, that get written to by a backend python process (mavlogd). Grafana is connected to the backend influxdb database where it gets all of it’s data. influxdb was chosen because it can store non-averaged data down at the nanosecond level.
The main work is parsing the log files and getting the data into influxbd. Once the data is there, you can query it at will with a rich query language as well as through language bindings. You can tag each logfile and then select the data by logfile, vehicle, time etc, which sounds close to what you’re looking for. Here’s the script that translates the log files into influxdb - it uses pymavlink dfreader for the underlying log reading/parsing:
ps - this has been my dilemma for the next step for this project. Tools like plot.dron.ee are cool because there’s essentially no installation needed - it’s all done in the client. However, you’re limited to client-side technology (which is improving by the day) and it’s much more difficult to do large scale analysis across flights/vehicles.
I’ve been trying to think of how to do a tool that will cater to both.
I normally use a github issue or gitter to discuss these kinds of things but they’re not very structured. Happy see whatever you have in your mind
Love your Ardupilog - I could see something similar in a high level python module onto eg. an influxdb backend with a predefined schema. That would be fab.
@Georacer Sorry for lack of response - been very busy and my todo list just seems to get ever longer! Hoping to sit down over the next couple of weeks and look at this closer, and perhaps thrash out a PoC. I do think quite a bit of this has already been done in Maverick, in some form or another.