Website: https://custom.ardupilot.org
Code: https://github.com/ArduPilot/CustomBuild
Introduction
As part of the GSoC 2021 programme, I set out to create a custom builder for ArduPilot firmware, with a vision of users having the option to create builds from any combination of vehicle, board, and a custom set of features.
A consequence of the broad functionality of ArduPilot firmware is that some software configurations can exceed 1MB, which may be larger than the flash memory capacity of some AutoPilot boards. The ArduPilot Custom Firmware Builder allows for users to have their desired features whilst minimising the flash memory usage.
The program is written primarily in python, using the Flask microframework to enable an interactive HTML website. The program consists of two main parts: the user interface and the backend.
User Interface
As shown in the first image, users can select a vehicle, a board and a set of features they want in their build, and the default selection of each can be set in the python script. The available features are displayed by category, with expandable menus for each category. Below the selection options is the build status table (second image), which displays all builds on the server from all users. Each item in the âStatusâ column links to the corresponding webpage for the build, and the âArduPilot Git Hashâ links to the GitHub commit of the ArduPilot repository used to create the build. This âhomeâ webpage is âindex.htmlâ, and the embedded status table is provided from âstatus.htmlâ.
The generate button then takes the user to the webpage for their build (âgenerate.htmlâ, as shown in the third image), which displays a real-time update of âbuild.logâ. This file displays the selected options and the progress of the build. It is also saved in the build directory on the server for access after the build has finished.
All of the HTML webpages use Jinja2 integration with Flask in order to display different data based on variables specified in the python program.
An Apache server with WSGI is used to both run the python program and to display the build directories, which allows users to download the .apj and .hex files of their build. This is shown in the fourth image below.
Backend
Flask
The Flask microframework allows for the core python program (âapp.pyâ), when run, to generate interactive HTML webpages, permitting data transfer between the user interface and the python program. Functions within the python program are called by either the current URL of the webpage, or the POST or GET HTTP methods.
Multiple Threads and Locking
The python program utilizes multiple threads (using the âthreadingâ library) in order to run multiple tasks simultaneously. Fetching data from the user input is done ordinarily every time a user visits the website and generates a build, however the build process itself and updating the status table are both continuously refreshed using two additional threads. All of the functionality mentioned later in the âBuilds managementâ section are run within these threads and are therefore continuously running as long as âapp.pyâ itself is running.
From the âthreadingâ library, locking is also implemented, which ensures certain processes run exclusively. This prevents potential errors occurring by two interdependent processes running simultaneously.
User Input
The list of boards is fetched from the ArduPilot repository every time the website is visited, and the list of available features is specified as a data class in the python script, which includes category, name, definition, display name and default (on or off). The vehicles are simply written out as a list in the python script.
After the user has selected the vehicle, board and features for their build and hit âGenerateâ, the âgenerateâ function is called, which initially creates âextra_hwdef.datâ containing the selected extra hwdef options. The md5sum of this file and the git hash of the current ArduPilot repository are concatenated with the vehicle and board name to form a unique âtokenâ for the build, and this token is used as the name of build directory. If the build already exists, or it is already queued, the âGenerateâ button directs the user to the pre-existing build directory, otherwise the directory is created and the âextra_hwdef.datâ file is removed from the parent âbuildsâ directory and written to the directory for the specified build. A âq.jsonâ file is also created from a dictionary, which contains information about the build: the token, source directory (ArduPilot repository), âextra_hwdef.datâ, vehicle, board, and IP address of the user. The ArduPilot repository and its submodules are also updated to begin with every time a user requests a build.
In summary, the build directory with the token as the directory name is created as soon as the build is requested, and it contains âextra_hwdef.datâ, âq.jsonâ, âbuild.logâ and âselected_features.jsonâ (discussed later) while it is in the build queue. Once the build starts running, the directory contains âbuild.logâ and âselected_features.jsonâ, and then also the completed build files once the build has finished.
Directories
The directory structure is as follows:
âCustomBuildâ is where âapp.pyâ and all the linked files exist. The working directories are in âbaseâ so that âCustomBuildâ is not accessible to the user, therefore minimising susceptibility to hacking. âardupilotâ is the latest GitHub commit of the ArduPilot repository, âbuildsâ is where the unique build directories are located, and âtmpâ contains temporary working files while the build is running. If it does not already exist, âbuildsâ is created as soon as âapp.pyâ is run and likewise âtmpâ is created when a build is run.
Builds Management
The creation and existence of the âq.jsonâ file in the build directory essentially adds the build request to the build queue, and once the build starts running, âq.jsonâ is removed and the build is therefore removed from the queue.
To provide the list of selected features and the short Git hash for the status table, âselected_features.jsonâ, which contains this information, is written to the build directory. Each row in the status table corresponds to a build directory, and the status of the build (âPendingâ, âRunningâ or âFinishedâ) is displayed depending on the existence of âq.jsonâ or certain indicators within âbuild.logâ. If the build has failed, the status table will display âFailedâ.
The age of each build directory is regularly checked and all directories (and therefore builds) older than 24 hours are deleted. This time frame could be altered in the future depending on server demand and storage space. The python program also sorts all the queued builds (build requests that have not run yet) from oldest to newest so that the oldest gets built first.
Preventing Denial of Service
As previously mentioned, the âq.jsonâ file contains the IP address of the user who requested the build. The program obtains a list of the addresses of all users who have requested a build, and then removes the directories of any build requests corresponding to an IP address in the list that already exists. This means that a user can only ever have one build in the queue, and if they request more builds, the program will remove their previous build request and only keep their most recent request, therefore preventing one user from dominating the server by submitting a large number of build requests in quick succession.
Running The Build
Within the build thread, the queued builds are run one at a time, from oldest to newest. The ârun_buildâ function takes arguments of the task file (âq.jsonâ), âtmpâ directory location, build directory location, and âbuild.logâ location. The waf configure and run build execute with the current working directory as the ArduPilot repository (source directory) and ââoutâ specified as âtmpâ for the temporary build files. The source directory location, âextra_hwdef.datâ, board and vehicle are all obtained from the task file. As the build runs, its progress is logged to âbuild.logâ.
Future Features
- GitHub tree selection
- Dependencies
- Automatically select features that user-selected features depend on
- Fill in a default set of features for any given board
- Calculate flash cost
- Approximate how much flash each feature uses, and use that to calculate and display remaining space left on a given board
- Provide the option to run the build using other ArduPilot Git branches
- For stable and beta branches
- For trusted developers, look for branches called custom-XXXX
- Display estimated time remaining for a build
- Based on time taken for previous similar builds
- Introduce a button in the status table which when pressed, replicates selections of that specific build
Many thanks to my mentor and other members of the ArduPilot community who have been very helpful during this project!
