How to export all log parameters from .bin file?

General
1

Hi all,
First time posting to this community, so apologies if this is not the right place to do so.

I am interested in exporting all the parameters from the .bin file into separate files, as I would like to upload and store this data in a central database (MS Access).
Currently, I was thinking about just looping through all params shown in the docs and using the pymavlink tools to export data. Currently, my code loops through all .bin files in a folder.

Log parameters page
https://ardupilot.org/copter/docs/logmessages.html

pymavlink command line tool:

param="BAT"
for fn in *.bin
do
    # echo "$fn"
    python "C:\path\to\script\mavlogdump.py" --format csv --types $param "C:\path\to\bin\files\\${fn}" > "C:\output\folder\path\${param}_${fn::-4}.csv"
done

Is there a better/more succinct way to accomplish this? Should I create an array of all desired params, and loop through those as well?

Thanks in advance!

2

If you’re just doing one or two log files, try this: Review the log file in Mission Planner. With the log file viewer open select the check box to “Show Params”. Then in the pop up that has the parameters “save to file”

3

image
image1016Ă—271 48.4 KB

Another possibility is to go to the DATAFLASH LOGS tab and select CREATE KML+gps. You will then be prompted to select a BIN file. In the same directory as the BIN file, a series of files will be created, including a parameter file.

4

Hi, @Lano @Allister Thanks for the replies. However, it does not seem like these are solutions for my particular application.
We have a fleet of drone pilots. Our idea is that, when they return to office, they would download the .bin files from the drone (to a usb). Then, go to our main server, plug in the usb, and use our script that would:

  • loop through each .bin file and export all parameter data (to json or csv files)
  • then upload those exported data files into MS Access database
  • more stuff down afterwards in script
1 Like
5

You do have the more scriptable/automatable solution. There is no better than that.

I do have a python script that is probably faster, but it needs adaptation for you because it checks out the default parameter values, not the parameter values. If you are interested the script is called extract_param_defaults.py and all you need to change is line 97 and 98 to extract Value instead of Default.

1 Like
6

You can simply use pymavlink to do this.

Here is a function that takes in the file path of the .bin file and returns a dictionary of the parameters:

def get_params(file_path):
    
    parameters = {"PARAM_NAME": [],
                 "Value": [],
                 "Default_Value":[]}
    
    connection = mavutil.mavlink_connection(file_path)
    m = connection.recv_match(type='PARM')
    while m:
        parameters["PARAM_NAME"].append(m.Name)
        parameters["Value"].append(m.Value)
        parameters["Default_Value"].append(m.Default)

        m = connection.recv_match(type='PARM')
    
    return parameters

you will have to import the mavutil.py from pymalink as shown below:

from pymavlink import mavutil

Further you can save these parameters into an file format as per your requirement.

7

@amilcarlucas @Suyash_Mali Thanks for the responses and sharing some code! I think both of your codes will work for my use case.

@Suyash_Mali Can you expand on what the Value and Default_Value params are? What gets populated there?
Does this function return what parameters there are in the .bin file, and then I would use another function to export the data from that dictionary?

8

It is the same code I use, but mine has export functions included

1 Like
9

The Value key contains the present value of the parameters, while the Default_Value contains the default values of the parameters that are set by ardupilot.

Yes, this function returns just a dictionary containing the Param Names, Their present values, and the default values of those parameters. You can use it according to your requirement.
And as said by @amilcarlucas, this function is just a striped-down version of his example.

10

@amilcarlucas thanks for your code, it is very helpful for learning to use pymavlink software. Currently, I am only getting the default value outputs, but I would like the actual measured data (i.e. the GPS time and location as a csv file). I changed the code to have:

if pname not in defaults and hasattr(m, 'Value'):
    defaults[pname] = m.Value

But this only returns a single value for each param.

In your code, how would you output each PARAM as a file, such as the output file generated from the code below:

python "C:\path\to\script\mavlogdump.py" --format csv --types GPS "2024-01-12 10-07-30.bin" > "GPS_OUTPUT.csv"

Edit:
I was able to output the entire file as a text using the following command:

python  "C:\path\to\script\mavlogdump.py" "2024-01-12 10-07-30.bin" > binoutput.txt

It seems like this doc describes the .bin file params well:
https://ardupilot.org/copter/docs/logmessages.html#logmessages

Would you recommend finding a way to loop through this output? Or can your code condense all the params together and output a file for each?

11

I have changed the tool to do what you want, but you need to get the new version from the GitHub - ArduPilot/MethodicConfigurator: A clear ArduPilot configuration sequence repository and then play with the command line parameters

12

Thanks for the quick response. For some reason, your code seems to only print the param_name and value, and not all the data. I was able to work with the pymavlink api, and it seems like I could pull each param using the:

m = mlog.recv_match(type=['GPS'])

Is there a list or function that returns all the types (which are the params in link below)?
https://ardupilot.org/copter/docs/logmessages.html#logmessages

13

Yes, for GPS messages you need to change the code in the way you described. You can add other message types to the list

14

Is this something I should just loop through a list of message types that I create manually?

15

No, you put them on on the list and loop only once.
The body of the loop is a switch-case similar statement that selects what to do depending on the message type

16

Hi @amilcarlucas @Suyash_Mali
I am unsure what happens with the code, but when I try to add more types (i.e. the GPS and a GPA), I only receive a few field names (not all of GPS and GPA).

m = mlog.recv_match(type=["GPS", "GPA"])
fieldnames = m.get_fieldnames()
print(fieldnames)

m = mlog.recv_match(type=["GPS"])
fieldnames = m.get_fieldnames()
print(fieldnames)

m = mlog.recv_match(type=["GPA"])
fieldnames = m.get_fieldnames()
print(fieldnames)

>> ['TimeUS', 'Status', 'GMS', 'GWk', 'NSats', 'HDop', 'Lat', 'Lng', 'Alt', 'Spd', 'GCrs', 'VZ', 'Yaw', 'U']
>> ['TimeUS', 'Status', 'GMS', 'GWk', 'NSats', 'HDop', 'Lat', 'Lng', 'Alt', 'Spd', 'GCrs', 'VZ', 'Yaw', 'U']
>> ['TimeUS', 'VDop', 'HAcc', 'VAcc', 'SAcc', 'YAcc', 'VV', 'SMS', 'Delta']

I use a while loop to get all data and save as json:

while True:
    print(m.to_dict())

Would you have any suggestions? Thanks

17

It might be a pymavlink but I think there might be an issue with your loop.

I use that code and it works.

18

Hi all,

Is there anyway to get a list of all parameters from ardupilot? Currently, I created a bash script to loop through params and export as json, but I would like to find a way to automate parameter extraction.
My goal is to dump all parameters into database from a fleet of drones we have.
code example:

for param in "ACC" "ATT" "BAT" "GPS" "GPA"
do
    for fn in *.bin
    do
        # echo "$fn"
        python "C:\path\to\pymavlink\Scripts\mavlogdump.py" --format json --types $param "C:\path\to\input\folder\\${fn}" > "C:\path\to\output\\${param}_${fn::-4}.json"
    done
done
19

Install ArduPilot Methodic Configurator and after that do

extract_param_defaults -t value binfile.bin

It will do what you want, and the ArduPilot Methodic Configurator can be used to manage the parameters of your drone fleet. It was designed to do that

20

@amilcarlucas thanks for sharing your software! It looks incredibly useful.
Would I just type in extract_param_defaults -t value binfile.bin in the terminal? I am looking for the simplest solution to integrate in our CI/CD pipeline for project management and data processing. E.g. when a pilot returns to our lab, they upload the bin file and all processing is done in scripts in the backend.