Servers by jDrones

Log database utility

database

(George Zogopoulos Papaliakos) #1

I imagine of a utility software which can act as a database for logs. Some of its features would be:

  • Database-only functionality: Its core is about storing and searching logs, not processing them.
  • Local storage: The logs are on your PC and databases are portable by physical means (USB sticks etc).
  • Log-agnostic: Not specific to ardupilot logs (dataflash or tlogs)
  • Metadata-based: Since any log file format is admissible, metadata fields contain vital information. Also useful to record external quantities (notes, pilot, platform, events, etc)
  • Extensible: It provides an API for implementing log-processing plugins.

Do you know if such a thing exists?
I know that a lot of web-based utilities exist but do not fit the requirements and most are closed-soure anyways.

If not, is anyone interested in discussing some specifications?


(Andras Schaffer) #2

It is called Mavrick analysis module :smiley:
https://goodrobots.github.io/maverick/current/#/modules/analysis


(George Zogopoulos Papaliakos) #3

I guess I’ll read the docs in detail at some point, but, I had the impression that Maverick is distributed as a complete system image.

Can you use the Analysis Module separately a-la python setup.py?


(Andras Schaffer) #4

I think it can be lifted easily from the image creation scripts…


(Fnoop) #5

Hey @Georacer, there are instructions here for running it without the rest of the maverick stuff:

Interested in what kind of thing you had in mind? I’m hoping to sit down in the next month or two and revisit logging. I was looking to do a client-only solution like the plot.dron.ee site (but open source), but there are lots of cool things you can do with a database-driven system like Maverick currently uses that you could never really do client-only.


(George Zogopoulos Papaliakos) #6

Here’s a little more info on what I have in mind:

I’m in the end of a PhD on UAV fault diagnosis and at some point I’d like to ask the community to voluntarily contribute logs for processing and analysis on common crash causes.

In support of this, I imagine a Python module for log management which one could import in order to pull in the log database functionality to your analysis front-end software.

To recap, what I think would be useful would be a (Python) library which would cover all of the features of the first post.

Possible applications would be:

  1. Plain log storage: I have a lot of logs and I need to go search and review them, similar to Dronee and other log book apps.
  2. Log analysis: More involved log examination algorithms, akin to Maverick and Log Analyzer.
  3. One-off research: You put a bunch of logs in one place (regardless of where you found them) and write your tailor-made analysis on the issues you are interested in, such as my case. For example, inferences on mean flight time across UAV classes, which APM version had the most crashes or (in my case) generate diagnostic signals and test their accuracy on real logs.

I imagine this as a vault, where one would put in logs, along with a bit of information, and the module would parse for rudimentary information. In addition to that, it would provide a useful API.

So, in summarily, I imagine this API kind of like:

Parsers:

  • Total number of logs
  • Total flight time
  • Number of different vehicles
    etc…

Getters (for any individual log):

  • Any quantity timeseries, properly timestamped
  • Metadata, useful for filtering and cross-correlation, as said before (UAV ID, battery ID, pilot, date, etc). This information could be stored in a document-based DB, such as TinyDB.
  • The whole log, for database migration but with some filters applied (possibly based on the aforementioned metadata, e.g. get me all the logs of this FW version).

I imagine specifying a directory under which you would create one folder for each log and one JSON file alongside it, containing the metadata.

Does all of this make sense?


(Fnoop) #7

@Georacer v.interesting!
So the Maverick analysis front-end is simply grafana with some custom dashboards, that get written to by a backend python process (mavlogd). Grafana is connected to the backend influxdb database where it gets all of it’s data. influxdb was chosen because it can store non-averaged data down at the nanosecond level.

The main work is parsing the log files and getting the data into influxbd. Once the data is there, you can query it at will with a rich query language as well as through language bindings. You can tag each logfile and then select the data by logfile, vehicle, time etc, which sounds close to what you’re looking for. Here’s the script that translates the log files into influxdb - it uses pymavlink dfreader for the underlying log reading/parsing:


(Fnoop) #8

ps - this has been my dilemma for the next step for this project. Tools like plot.dron.ee are cool because there’s essentially no installation needed - it’s all done in the client. However, you’re limited to client-side technology (which is improving by the day) and it’s much more difficult to do large scale analysis across flights/vehicles.
I’ve been trying to think of how to do a tool that will cater to both.


(George Zogopoulos Papaliakos) #9

I intended to create specs/structure document to gradually flesh out this log database module, in something like Google docs or Dropbox paper.

Do you have a favourite format for this kind of thought sharing, so that you can see what I have in mind and maybe talk more in-depth?


(Fnoop) #10

I normally use a github issue or gitter to discuss these kinds of things but they’re not very structured. Happy see whatever you have in your mind :slight_smile:
Love your Ardupilog - I could see something similar in a high level python module onto eg. an influxdb backend with a predefined schema. That would be fab.


(George Zogopoulos Papaliakos) #11

Okay, so this is a first draft:

I’ll flesh it out more in the following days.

Tell me if you can view, post-comments or edit this.

Thanks


(George Zogopoulos Papaliakos) #12

Viewing is open for everyone. Editing and commenting requires Dropbox login.


(George Zogopoulos Papaliakos) #13

@fnoop and others,
I believe I have a rough draft, which carries out the concept well, nonetheless.
See the previous Dropbox Paper file.

I am open to comments and discussions.


(Fnoop) #15

@Georacer Sorry for lack of response - been very busy and my todo list just seems to get ever longer! Hoping to sit down over the next couple of weeks and look at this closer, and perhaps thrash out a PoC. I do think quite a bit of this has already been done in Maverick, in some form or another.


(George Zogopoulos Papaliakos) #16

You and me buddy… No worries.

I’ll definitely take peaks from your Maverick implementation of influxDB and log loading, but I really need a standalone Python library to interface with it, possibly through Pandas frames.

If you really have done everything I need in Maverick, I’ll just port it to a library and be done with it! :wink: