Servers by jDrones

GSoC 2021 - Custom Firmware Builder

Website: https://custom.ardupilot.org
Code: https://github.com/ArduPilot/CustomBuild

Introduction

As part of the GSoC 2021 programme, I set out to create a custom builder for ArduPilot firmware, with a vision of users having the option to create builds from any combination of vehicle, board, and a custom set of features.

A consequence of the broad functionality of ArduPilot firmware is that some software configurations can exceed 1MB, which may be larger than the flash memory capacity of some AutoPilot boards. The ArduPilot Custom Firmware Builder allows for users to have their desired features whilst minimising the flash memory usage.

The program is written primarily in python, using the Flask microframework to enable an interactive HTML website. The program consists of two main parts: the user interface and the backend.

User Interface

As shown in the first image, users can select a vehicle, a board and a set of features they want in their build, and the default selection of each can be set in the python script. The available features are displayed by category, with expandable menus for each category. Below the selection options is the build status table (second image), which displays all builds on the server from all users. Each item in the ‘Status’ column links to the corresponding webpage for the build, and the ‘ArduPilot Git Hash’ links to the GitHub commit of the ArduPilot repository used to create the build. This ‘home’ webpage is ‘index.html’, and the embedded status table is provided from ‘status.html’.


The generate button then takes the user to the webpage for their build (‘generate.html’, as shown in the third image), which displays a real-time update of ‘build.log’. This file displays the selected options and the progress of the build. It is also saved in the build directory on the server for access after the build has finished.

All of the HTML webpages use Jinja2 integration with Flask in order to display different data based on variables specified in the python program.

An Apache server with WSGI is used to both run the python program and to display the build directories, which allows users to download the .apj and .hex files of their build. This is shown in the fourth image below.

Backend

Flask

The Flask microframework allows for the core python program (‘app.py’), when run, to generate interactive HTML webpages, permitting data transfer between the user interface and the python program. Functions within the python program are called by either the current URL of the webpage, or the POST or GET HTTP methods.

Multiple Threads and Locking

The python program utilizes multiple threads (using the ‘threading’ library) in order to run multiple tasks simultaneously. Fetching data from the user input is done ordinarily every time a user visits the website and generates a build, however the build process itself and updating the status table are both continuously refreshed using two additional threads. All of the functionality mentioned later in the ‘Builds management’ section are run within these threads and are therefore continuously running as long as ‘app.py’ itself is running.

From the ‘threading’ library, locking is also implemented, which ensures certain processes run exclusively. This prevents potential errors occurring by two interdependent processes running simultaneously.

User Input

The list of boards is fetched from the ArduPilot repository every time the website is visited, and the list of available features is specified as a data class in the python script, which includes category, name, definition, display name and default (on or off). The vehicles are simply written out as a list in the python script.

After the user has selected the vehicle, board and features for their build and hit ‘Generate’, the ‘generate’ function is called, which initially creates ‘extra_hwdef.dat’ containing the selected extra hwdef options. The md5sum of this file and the git hash of the current ArduPilot repository are concatenated with the vehicle and board name to form a unique ‘token’ for the build, and this token is used as the name of build directory. If the build already exists, or it is already queued, the ‘Generate’ button directs the user to the pre-existing build directory, otherwise the directory is created and the ‘extra_hwdef.dat’ file is removed from the parent ‘builds’ directory and written to the directory for the specified build. A ‘q.json’ file is also created from a dictionary, which contains information about the build: the token, source directory (ArduPilot repository), ‘extra_hwdef.dat’, vehicle, board, and IP address of the user. The ArduPilot repository and its submodules are also updated to begin with every time a user requests a build.

In summary, the build directory with the token as the directory name is created as soon as the build is requested, and it contains ‘extra_hwdef.dat’, ‘q.json’, ‘build.log’ and ‘selected_features.json’ (discussed later) while it is in the build queue. Once the build starts running, the directory contains ‘build.log’ and ‘selected_features.json’, and then also the completed build files once the build has finished.

Directories

The directory structure is as follows:

‘CustomBuild’ is where ‘app.py’ and all the linked files exist. The working directories are in ‘base’ so that ‘CustomBuild’ is not accessible to the user, therefore minimising susceptibility to hacking. ‘ardupilot’ is the latest GitHub commit of the ArduPilot repository, ‘builds’ is where the unique build directories are located, and ‘tmp’ contains temporary working files while the build is running. If it does not already exist, ‘builds’ is created as soon as ‘app.py’ is run and likewise ‘tmp’ is created when a build is run.

Builds Management

The creation and existence of the ‘q.json’ file in the build directory essentially adds the build request to the build queue, and once the build starts running, ‘q.json’ is removed and the build is therefore removed from the queue.

To provide the list of selected features and the short Git hash for the status table, ‘selected_features.json’, which contains this information, is written to the build directory. Each row in the status table corresponds to a build directory, and the status of the build (‘Pending’, ‘Running’ or ‘Finished’) is displayed depending on the existence of ‘q.json’ or certain indicators within ‘build.log’. If the build has failed, the status table will display ‘Failed’.

The age of each build directory is regularly checked and all directories (and therefore builds) older than 24 hours are deleted. This time frame could be altered in the future depending on server demand and storage space. The python program also sorts all the queued builds (build requests that have not run yet) from oldest to newest so that the oldest gets built first.

Preventing Denial of Service

As previously mentioned, the ‘q.json’ file contains the IP address of the user who requested the build. The program obtains a list of the addresses of all users who have requested a build, and then removes the directories of any build requests corresponding to an IP address in the list that already exists. This means that a user can only ever have one build in the queue, and if they request more builds, the program will remove their previous build request and only keep their most recent request, therefore preventing one user from dominating the server by submitting a large number of build requests in quick succession.

Running The Build

Within the build thread, the queued builds are run one at a time, from oldest to newest. The ‘run_build’ function takes arguments of the task file (‘q.json’), ‘tmp’ directory location, build directory location, and ‘build.log’ location. The waf configure and run build execute with the current working directory as the ArduPilot repository (source directory) and ‘–out’ specified as ‘tmp’ for the temporary build files. The source directory location, ‘extra_hwdef.dat’, board and vehicle are all obtained from the task file. As the build runs, its progress is logged to ‘build.log’.

Future Features

  • GitHub tree selection
  • Dependencies
    • Automatically select features that user-selected features depend on
  • Fill in a default set of features for any given board
  • Calculate flash cost
    • Approximate how much flash each feature uses, and use that to calculate and display remaining space left on a given board
  • Provide the option to run the build using other ArduPilot Git branches
    • For stable and beta branches
    • For trusted developers, look for branches called custom-XXXX
  • Display estimated time remaining for a build
    • Based on time taken for previous similar builds
  • Introduce a button in the status table which when pressed, replicates selections of that specific build

Many thanks to my mentor and other members of the ArduPilot community who have been very helpful during this project!

39 Likes

It has been a real pleasure working with Will on this project! Really nice work and very useful result

9 Likes

This really is a very important development. Well done!

4 Likes

Thanks a lot, great work!

1 Like

This is awesome!

Will be really great to get a version or hash/branch drop down to make the build more specific to a version.

I assume now its just using the newest release?

2 Likes

Yes for now it’s just the latest release, but an option to use different branches is a high priority for future features.

1 Like

That’s awesome… well done

1 Like

As someone who still uses almost exclusively 1MB boards, I can’t stress enough how helpful this is - great work, thank you!

2 Likes

Wow, that’s really helful

1 Like

this great feature. thank you

This is great!

I wonder, if it will be possible to activate the Lua scripting feature for 1M boards like Pixhawk1-1M or OmnibusNanoV6.
A basic Rover build should have space for scripting.

Is there an ability to modify the hwdef? For example, I’m using the RX_In on a PixRacer Pro as LED out. Thanks!

Suggestion: It would be great if the user could also decide on a font when compiling. That way there would be no need to put additional fonts on sd card for those who don’t like the default one.

No, HWdef modifications like that need to be done by the user

Servers by jDrones