Servers by jDrones

Converting Bin files from PixHawk to CSV?

I am looking for a python script or anything at this point for a method to parse the data from a pixhawk that was flashed with mission planner firmware.(That is the extent I know of the hardware of it I do apologize) The file is a bin file format and was pulled from the SD card. I want to be able to analyze this data using Python. I have very limited knowledge of python, however, enough to know how to grab data from a CSV file and run some plot analysis.

I was hoping there would be any scripts written that would do this job? Any help pointing me to the right direction would be much appreciated.

Thank you.

I believe this is what you are looking for: https://github.com/ArduPilot/pymavlink/blob/7b0d51cca7e75b3cf84f5dbb74e76f727816e50d/tools/mavlogdump.py

Alternatively, Misison Planner, under the “Dataflash” tab, has converters to CSV and MATLAB formats.

Are you sure Dataflash tab has a convert to CSV option? I only have 6 options:
Download Data flash Log via Mavlink
Review a Log
Auto Analysis
Creat KML+GPX
PX4 bin to log
Create matlab file
I don’t see anything for converting to CSV. I do see it in telemetry logs convert to CSV but I don’t have any tlogs only have bin files from sd card downloads.

As for the mavlogdump.py, Could you help me try to run it? I have tried to run it but for some reason it generates an empty file. What command do I need to run?

Try pressing the ‘PX4 bin to log’ button. I think this is what you want.

Yes that converts it to a csv file, however, it has not been parsed. I am wondering does the script you linked does a better job at parsing? I am having issue running it because I can’t clone mavutil correctly.

What do you mean by “It has not been parsed”? Perhaps you have some other, specific form of CSV on your mind?

Regarding mavlogdump, I noticed that you must specify the message types you want manually, when selecting CSV, so I don’t recommend it.

Well, the format that I see the csv file is something like this:

IMU, 487293681, 0.3007393, 0.114936, -0.04736611, -0.379998, 0.07098468, -13.61945, 0, 0, 47.6759, 1, 1
IMU2, 487293681, 0.2675287, 0.1225688, -0.05286679, -0.3815768, 0.1907908, -13.32735, 0, 0, 50.375, 1, 1
GPS, 487294117, 4, 408521000, 1919, 12, 0.78, 34.6714628, -118.3349373, 921.36, 20.62035, 359.6666, 2.96, 1
GPA, 487294117, 0.97, 0.98, 1.38, 0.52, 1, 487294
MAG, 487294266, 428, 210, 274, -115, 0, 1, 0, 0, 0, 1, 487294257
MAG2, 487294266, 306, 129, 354, 77, -135, 38, 0, 0, 0, 1, 487294259
ARSP, 487294315, 17.68646, 156.9075, 29.63, 156.9075, 101.5598, 1
BARO, 487294384, 67.95848, 91898.86, 49.00, -4.158154, 487294, 0
CURR, 487294444, 15.60016, 5.479106, 321.8991
POWR, 487294454, 5.212646, 5.793, 1
SONR, 487294505, 0, 0, 0, 0
IMU, 487293681, 0.3007393, 0.114936, -0.04736611, -0.379998, 0.07098468, -13.61945, 0, 0, 47.6759, 1, 1
IMU2, 487293681, 0.2675287, 0.1225688, -0.05286679, -0.3815768, 0.1907908, -13.32735, 0, 0, 50.375, 1, 1
GPS, 487294117, 4, 408521000, 1919, 12, 0.78, 34.6714628, -118.3349373, 921.36, 20.62035, 359.6666, 2.96, 1
GPA, 487294117, 0.97, 0.98, 1.38, 0.52, 1, 487294

As you can see IMU shows up every time there is a new data. I am very new to python so parsing this would be pretty hard for me. Usually I am use to data that is the same, for example IMU, will be all under one column.

Also I found another mavlogdump.py somewhere no idea where but the code looks like this. Not sure if this is the same As the one you linked.
#!/usr/bin/env python

'''
example program that dumps a Mavlink log file. The log file is
assumed to be in the format that qgroundcontrol uses, which consists
of a series of MAVLink packets, each with a 64 bit timestamp
header. The timestamp is in microseconds since 1970 (unix epoch)
'''

import sys, time, os

# allow import from the parent directory, where mavlink.py is
sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(__file__)), '..'))

import mavutil

from optparse import OptionParser
parser = OptionParser("mavlogdump.py [options]")

parser.add_option("--no-timestamps",dest="notimestamps", action='store_true', help="Log doesn't have timestamps")
parser.add_option("--planner",dest="planner", action='store_true', help="use planner file format")
parser.add_option("--robust",dest="robust", action='store_true', help="Enable robust parsing (skip over bad data)")
parser.add_option("-f", "--follow",dest="follow", action='store_true', help="keep waiting for more data at end of file")
parser.add_option("--condition",dest="condition", default=None, help="select packets by condition")
(opts, args) = parser.parse_args()

if len(args) < 1:
    print("Usage: mavlogdump.py [options] <LOGFILE>")
    sys.exit(1)

filename = args[0]
mlog = mavutil.mavlink_connection(filename, planner_format=opts.planner,
                                  notimestamps=opts.notimestamps,
                                  robust_parsing=opts.robust)

while True:
    m = mlog.recv_match(condition=opts.condition, blocking=opts.follow)
    if m is None:
        break
    if opts.notimestamps:
        print("%s" % m)
    else:
        print("%s.%02u: %s" % (
            time.strftime("%Y-%m-%d %H:%M:%S",
                          time.localtime(m._timestamp)),
            int(m._timestamp*100.0)%100, m))

The MAVlink protocol has about 100 different messages in use right now, each with 3-15 fields each. Moreover, some contain ints, floats or strings. What is more, each message is published in different order and frequency.

You cannot have a meaningful representation where each line will contain one instance of a fixed set of messages, aligned in columns. What you ask is problematic and out of the scope of this message protocol, in my opinion.

Perhaps you could process the data more intuitively in MATLAB, with the .mat file extension. In that form, you have each message in a separate structure variable, timestamped.

Other than that, I think the .csv you have in your hands is as clean data as you can get.

P.S. That mavlogdump.py seems to be older that the one I linked.

You cannot have a meaningful representation where each line will contain one
instance of a fixed set of messages, aligned in columns. What you ask is
problematic and out of the scope of this message protocol, in my opinion.

I’ve been considering this over the last few days. I was wondering about
allowing full specification of fields when doing a CSV dump; the “–types
FOO” would give you a “beat” at which all of your specified columns would
be emitted (–csv-columns SERVO_RAW_OUTPUT.servo3_raw,ATTITUDE.pitch)

Any thoughts?

Hi!
I am interested to use this file, but I can’t understand the input syntax to get .csv file.
When I type, for example
python mavlogdump.py -o temp.csv --planner --format csv --types ATT log001.bin
I get the data printed into the terminal, but the output file is still in bin format
Thanks!

Use that:

python mavlogdump.py --planner --format csv --types ATT log01.bin > temp.csv

I had the same problem a few days ago, hope this helps (convert the binary file to log file first, if there are any errors let me know)
log2csv.txt (3.2 KB)

Servers by jDrones