Develop/deploy a web-based LOG visualiser for helping user/s [Withdrawn]

Proposal type: Hardware [ ] , Software [ X] , Other [ ] : _________________

Description: We need a website where people can upload their log, and get basic information about it, with some graphs, maps, visualizations etc.
eg: droneshare used “doorama” for doing fly-throughs.
something equivalet to “droneshare” or px4s “plot_app”

Planned amount $$ (USD): 2000 ?

Estimated time for completion: 3 months duration.

helpful links:
droneshare/s javascript “frontend” is 100% open already: https://github.com/dronekit/droneshare ( it’s REST based backend is not )
px4 visualiser is also open:
code: https://github.com/PX4/flight_review
live example: http://review.px4.io/plot_app?log=10b69a65-c56e-476c-9d60-dbf101911ab8

Hi,
I have noticed that this proposal has been identified as a potential project in the 2017 google summer of code (awesome). I have been spending a bit of time recently integrating some of my prior log processing tools with a Flask (python) front end and a Bokeh (python / javascript back end). The current code is hosted on an extremely under powered server at hawkview.io. The project aims to replicate the core functionality of MAVExplorer but with a web interface… I was going to hold off posting anything with regards to this project until it was somewhat more mature and useful, but seeing as it might be useful to some one working on the problem (e.g.: google summer of code), all my work to date can be found in my GitHub repository here.

Additional items I would like to add / see added:

  • logged in views for users
  • user customisable plots
  • data reduction by flight mode and conditions (mask numpy arrays with NaN values to inhibit display in Bokeh)
  • automatic fault detection
  • fft plots
  • optional log obfuscation (location) for shared logs
  • Support for MAVCesium style 3D visualisation playback
  • emails to notify users that their log has been processed
  • support for remote upload from companion computers via APSync or similar
  • etc…

I’ll still keep this project moving forward, but the code is there for anyone who wants to use it support of the Ardupilot community.

Thanks,
Sam

I had a quick look at your work ( hawkview.io ), tried uploading a random log , etc, and it looks great.

nice, thanks Samuel!
This looks like it is python based, and does server side log processing?
For the GSoC I was hoping someone could do a version where the log parsing and display is in JavaScript running in the users browser. Otherwise I am concerned that if it becomes too popular the load on the server could become too high.
I did something similar in JavaScript many years ago for my solar system:
http://solar.tridgell.net/live/
that parses csv files from my inverters and displays graphs in JS, with no load on the server. I don’t really know if its practical to do this with large DF logs in JS, but I suspect it is.

Hi @DavidBuzz and @tridge,
Thanks for having a look. The plotting side of the application still needs a lot of work to be useful but the fundamentals of uploading and processing a log are pretty solid. The log processing and plot session hosting does occur server side in this application. As you suggested, server resources could become an issue if the number of users performing large amounts of analysis on big logs was high. I have attempted to minimise server CPU and RAM usage by trying to be clever with how the application works (which I will cover at the end of this post if you are interested…).

@tridge Your suggestion of a client side processing and log display + interaction is a really good idea for a lightweight, almost server-less implementation (having the server only serve the javascript, which then does all the work client side). That form of implementation does have its own set of limitations when it comes to being able to share logs and analysis, but it really comes down to what service(s) Ardupilot wants to offer its user base. Sorry for adding confusion re the GSoC objectives, that was not my intention!

I’d be interested to know if a refined version of this application would be at all useful to the Ardupilot community, and if so, I would happily take on ideas to help meet desired requirements and features. On the flip side if it is not envisioned to be of use that’s also perfectly fine, as I’m enjoying using the project to teach myself some new python web technologies :slight_smile:

Some notes on how the application works and where / how I have tried to save server memory:
Uploaded files are placed in a REDIS queue to be processed by a single celery background task. By limiting the background task to one worker, logs are processed sequentially in the order they were uploaded. e.g.: If multiple users upload multiple logs the lone processing task will only ever consume as much memory as the size of the log it is currently processing.

The heavy lifting of the processing occurs in a modified version of mavmemlog, which writes the log to disk in a directory structure. All messages of each type are stored in binary numpy arrays with named columns, while other useful bits of data (such as flight modes, params, etc…) are later stored in JSON text files.

Attacking the problem this way means each log is only ever processed once and it allows the required message types of interest (now binary numpy arrays with named columns) to be loaded very quickly from file. This significantly reduces memory requirements during the plotting operation and makes math / conditional masking easy.

Lastly, I was having issues with the bokeh plotting servers not releasing memory correctly so I wrapped the server and added a (currently very simple) memory watching load balancer which only spins up a new bokeh server when a user starts an analysis on a log. If the session times out or the user navigates away from the analysis page the bokeh server connection is closed, the process stopped and it re-enters the available server pool.

Cheers,
Sam

yes, I for one am definately interested, and I’m sure we can fund the server resources.

it isn’t just load, its also usability for the user. For example, when I analyse a log I commonly zoom into one critical region less than 1 second long, and I need to see every data point we have in that second (eg. 400 values). With server side processing you can’t produce graphs that detailed for the whole log, so you can only do big-picture views.
With a client-side viewer you can zoom right down to the individual data points.

The client side JS code should be able to access any log on the server. So we’d still have a common server where users upload logs, its just that the graph rendering would be done client side.
The JS I used for my solar system is here:
http://solar.tridgell.net/live/graphs.js
I think it wouldn’t be too hard to adapt this to DF logs.
Another concern I have with server side is the potential for security vulnerabilities. We’d need to make sure all the log processing couldn’t be manipulated by a malicious user uploading a bad log to either breach or crash the server.
A server side solution is still better than having no solution at all, but longer term I’d much prefer a client side system.
Cheers, Tridge

http://plot.dron.ee currently offers most of the features we were looking for, it’s closed source, doesn’t do 3D visualisations, doesn’t have a back-end that lets ardupilot do statistical analysis later, doesn’t let users “share” their logs easily with someone else ( ie upload to a unique url for sharing etc ) , but otherwise it’s javascript in-the-browser log analysis is fast and its graphs are good.