Resource organization for building firmware

I am trying to learn to work with ardupilot code, in particular APMrover2 on a Pixhawk platform, controlling a skid-steer ArduBoat. I want to get beyond fiddling with parameters and on to experimenting with some deeper changes.

I’m currently trying to work with Eclipse on a Windows XP machine, but after months of earnest effort the maze of layers and wrappers (and the moving-target aspect of the chase) has just about got me beat. I haven’t even been able to build the .px4 firmware from unmodified source code, neither via “make px4-v2” in a MinGW32 console from the appropriate vehicle directory, nor via Eclipse.

I think I may not have the resource dependencies organized correctly. Currently I have this pixhawk_toolchain directory tree. (It’s zipped 'cause it would be 2.5MB as .txt file.)

When I command “make px4-v2” from my C:/pixhawk_toolchain/Firmware/ardupilot/APMrover2 directory, it returns this error.

I have previously commanded “git submodule init” from C;/pixhawk_toolchain/Firmware, which appears to successfully register 14 submodules, but these do NOT include the 7 submodules inside C:/pixhawk_toolchain/Firmware/ardupilot/modules, viz. gbenchmark, gtest, mavlink, PX4Firmware, PX4NuttX, uavcan, and waf (which I placed there manually).

I’d just as soon get away from waf and git at least until I can get a handle on how the layers and dependencies in the tangle of source code for Pixhawk get linked and compiled to build a 700kB .px4 executable out of 2GB of resources!

If anyone, who has successfully compiled any vehicle source code for Pixhawk platform, can clue me in on their working code tree, I’d sure appreciate the insight.

Cheers!

You need to use “git submodule update - - init - - recursive” to load all the submodules.

I have built code on windows and on Ubuntu.

I followed these steps: http://ardupilot.org/dev/docs/building-px4-for-linux-with-make.html

Mike

@martinandriener

Really this stuff is much easier on Linux, but

It is easier to use git and to use the git clone command rather than downloading the zip.
Assuming that you have a valid compiler in your path e.g . gcc-arm-none-eabi-4_9-2015q3/bin

Once you have cloned the Ardupilot repo ( I would suggest starting again in a clean parent directory), then you should update the submodule ( git submodule etc) from within the main ardupilot directory, not the parent directory. After that, assuming your arm gcc compiler is in your path, then entering your vehicle directory and invoking make for your target should work ok.

Note that the devs are very keen on waf. For example the examples now can only be built with waf and I think that will soon be true for vehicles In fact after the first build, the waf build is much quicker if you touch a source file so it is recommended over make.

Thank you both, @iseries and @skyscraper, for your replies. I have pored over the instructions at the page you referred to, Mike, as well as the Building ArduPilot for Pixhawk/PX4 on Windows with Make page and the Building for Pixhawk/PX4 using Eclipse on Windows page.

The g++ compiler I’m using is evidently at my c/pixhawk_toolchain/toolchain/bin/g++.exe, as indicated by a “which g++” query in the MinGW32 console. I see there is also one at c/pixhawk_toolchain/toolchain/arm-none-eabi/bin/ (which I added to the Windows “path” variable but I don’t think it matters in light of the "which g++ result).

I’m certainly not facile with git, but I have run “git submodule update --init --recursive” from my c:/pixhawk_toolchain/Firmware directory; this apparently registered 20 submodules, but failed to recurse into 4 submodule paths, as indicated by this return.

Seven git submodules in the path c:/pixhawk_toolchain/Firmware/ardupilot/modules, namely “gbenchmark”, “gtest”, “mavlink”, PX4Firmware", PX4NuttX", “uavcan”, and “waf” were not apparently registered, though they are listed in the .gitmodules file in c:/pixhawk_toolchain/Firmware/ardupilot.

If I do “git submodule update --init --recursive” from within the main ardupilot directory, c:/pixhawk_toolchain/Firmware/ardupilot, it returns “You need to run this command from the toplevel of the working tree”, which I take to be c:/pixhawk_toolchain/Firmware.

I am working with directories downloaded as zip because the Windows px4 toolchain Software Download tool failed to git clone the ardupilot directory, and it seemed best to assemble manually everything necessary to build code for pixhawk.

I understand that the directory organization would be critical, and that is why I included the link to the pixhawk_toolchain directory tree that I am currently working with (generated by “tree /F /A > tree_pixhawk_toolchain.txt” as a Windows command). I was kind of hoping someone who had successfully built ardupilot code could reciprocate, for the sake of my understanding “under the hood”, rather than just getting it to work.

Having some familiarity with unix and similar environments I guess I’m not terribly averse to regrouping and starting again with Linux. In any case I’d sure like to be able to devote more efforts to working with control algorithms and C++ coding rather than just code repository management tools!

Is it not possible to revert to building (a snapshot of) ardupilot code without the developer synchronization tools, without internet connection (once all the libraries are once fetched)?

Thanks!
Pete

Are you running git in GitBash or Bash as supplied with Cygwin, if not it won’t work AFAIU.

(GitBash is actually a customized install of CygWin)

If you like to use Linux, save yourself a world of pain (or macOS)

I’ve never understood why Microsoft has never come up,with a ‘brew’ ‘macports’ or apt or yum like feature. It really needs it!

Ps you may want to look at the new build tools as well http://ardupilot.org/dev/docs/building-the-code.html

Make is being deprecated

hey martinandriener

I guess I might help you a bit, I’m on the same boat called windows. I’m not with Rover but Copter, and am on Win7 and not WinXP, but my procedure is a s follows

  1. first, and most importantly, you should know these wiki pages:
  1. download pixhawk_toolchain_installer_latest.exe and run it
    this is actually quite a nice piece, since it installs everything but doesn’t hurt any other installation on your PC, so, if you have e.g. an arm-eabi somewhere else, or python, or whatever, this is not affected and no conflict happens !
    there is a bit of a downside in that it now makes you installing all drivers, and doesn’t give you the option to skip that part as in older versions, which is nasty since it takes some time, and I never had found these drivers to work out for me (e.g. make-upload doesn’t work), however, I also never found these drivers to hurt, so, no need to worry here
    there is also a bit of downside in that it is totally unclear which version the pixhawk_toolchain_installer is, so you essentially have to reinstall it often
    otherwise it’s really a nice piece, since totally dependence free
    so, JUST RUN IT, say always yes to everything, and you get all you need

  2. get the code
    this step depends on which tools you’re using, I’m using Github Desktop, and not Git for Windows as mentioned in the wiki. Github Desktop makes it quite easy, but needs maybe a bit of getting used to it. Anyway, with your git-based tool, clone the master repository, and check out the branch you want (master, 3.4., or whatever), you should then have a github folder on your drive with an ardupilot subfolder.
    As regards downloading the zip: In my “early” days I also did it by downloading the .zip, and it worked fine back then, but I would NOT recommend that anymore! The reason is that they are now heavily using git submodules (or however that is correctly called in git-language), and to get the code of those is going to be a real pain! This is where I stumbled and which made me using a git-based tool to get the firmware folders from the git repository.
    So, DON’T WASTE YOUR TIME with zips, you’re much better off using one of the two mentioned git things.

  3. get the submodules
    this step also may depend on the git tool you’re using
    Github desktop does it all for you automatically, i.e. you don’t have to do this step, for other tools I don’t know.
    With Github Desktop it would manually go as this:
    open GitShell, browse to the git firmware folder, e.g. github\ardupilot, and enter git submodule update --init --recursive
    your git tool should also offer you a command line thingy, where to enter git submodule update --init --recursive

  4. copy and “initialize” the folders
    I’m not developing using git, and want code organized “my way”, so what I do is to copy the ardupilot folder to somewhere else and rename it, let’s say blabla/xpilot. This will break the git, and it will make the px4 toolchain to not work. So, in the blabla/xpilot folder, what I do is this:
    delete the .git and .github subfolders
    open GitShell, and browse to the blabla/xpilot folder
    run git init
    run git add *
    run git commit -m "any comment you like"
    this corrects the git

  5. setup Eclipse
    in step (2) you should have gotten a PX4 Toolchain entry in the Start Menu, go there, and run PX4 Eclipse
    when, follow the f… manual: http://ardupilot.org/dev/docs/editing-the-code-with-eclipse.html#editing-the-code-with-eclipse
    I do the “Creating the Project from scratch” thing
    the only thing I do differently is the first step, where you set up the new project. Instead of the mentioned name&folder, I enter whatever name I find convenient, and select the “…\blabla\xpilot” subfolder, and NOT the vehicle subfolder. The advantage is that I then also can easily browse to e.g. the libraries, look into the code of other vehicles, and so on.
    Note that in the later step, where you enter the build folder, you MUST use the vehicle subfolder, as told in the wiki.

That’s it. This was a lot of text, my memo sheet looks much shorter, as this
clone into GitHub
open Git Shell, git submodule update --init --recursive
copy to desired folder, rename
delete .git and .github
open Git Shell, git init, git add *, git commit -m "whateveryouwant"
setup Eclipse: http://ardupilot.org/dev/docs/editing-the-code-with-eclipse.html#editing-the-code-with-eclipse

This at least works fine for me.
Have fun and enjoy it for as long as it lasts, my extrapolation is that Windows folks won’t be able to compile ArduPilot in a near future.

Olli

I see there are a lot of other useful answers now, but just on one point.
arm-none-eabi-g++ is the compiler that is used to build the pxf version of ardupilot. rather than your system g++
There is no collision I think since it is invoked by the build system as arm-none-eabi-g++ rather than g++

It also sound like there are issues the way you have your tree set up, since I think ardupilot should be the ‘top level’

Anyway I hope you get this sorted. It is possible just a case of figuring out what is the problem :slight_smile:

Windows XP

I really don’t think you are going to have success on XP, you may, but Windows7 is really a minimum in this case.

@billb, @skyscraper, and especially @olliw42, I greatly appreciate your thoughts on this and the time that you took to compose helpful responses.

As you get my drift, I’m as much interested in understanding more deeply how the build (compiling/linking) process works, even how it could alternatively work, as I am interested in just getting something to work. (One point I’m specifically interested in clearing up is whether the “ardupilot” directory belongs inside the “Firwmare” directory or vice-versa.)

I’m not quitting on Windows XP just yet as I reckon that with MinGW32 and all the tools that are part of the v16 pixhawk toolchain one ought to be able to build the source code if one fully understands all the linking. But I’m convinced to get a linux machine going as well.

It won’t be the first regrouping/digression for me; About a month and a half in September and October were devoted to studying Lippman, Lajoie and Moo C++ Primer along with Stroustrup’s C++ bible, since my previous programming experience had been in C. Maybe I also need to take the next step in remedial education and study more on the general topic of code-building.

Whew, I so regret not having gotten into the ardupilot game about 2009. I’m seriously considering rewriting Pixhawk-platform-specific code in a way that I can manage, especially since I aim to use true heading data from gyro (actually GPS-based), input to serial port, rather than using heading derived from magnetometer on I2C bus. But at the end of the day I don’t want to lose MAVlink; I find Mission Planner to be an extremely useful development tool.

I don’t aim to reinvent wheels, but I sense the need (at least with my limited mental resources) to work with a less universal code base, and probably not within the playground of developers who are really in a different league than I.

Anyway, I’ll let you know how it goes as I follow up on good suggestions.

Cheers!
Pete

To understand the build process for px4, you could try looking at and working on https://github.com/PX4/Firmware . That is the PX4 library part of the px4 ArduPilot build and can be built separately. (Note that the ArduPilot version may not be compatible with the official px4 , since I think various patches are added). If you watch the build of that it should be familiar to the px4 Ardupilot build.

The other great way to understand the buils process is to capture the output to a file. There is a verbose option somewhere in the makefile which shows more of the actual compiler output, which by default is suppressed.
From memory, if you add a VERBOSE=True or similar to your command line (or just hack the makefile) it will show more output.
see
https://github.com/ArduPilot/ardupilot/blob/master/mk/sketch_sources.mk#L85

Re; learning C++. I started from Bjarne Stroustrup’s book and to me it was like coming home. C++ is my number 1 language. I think it is worth persevering with, but I guess I am biased.

Hi Pete,

Windows is a complicated developer environment but, as the lead dev of Copter uses it, we try to support it the best we can.
My recommendation is to install the latest Windows Pixhawk Toolchain and use a Git tool (possibly GitHub for Windows) to clone the ArduPilot repository. I think that using zips of the repository isn’t well supported at this time.
I would also clone the repository to a path outside the path of the toolchain.

If you need more help you may also go to our Gitter chat (http://gitter.im/ArduPilot/ardupilot). Interaction in real-time sometimes is more helpful.

I did succeed in building (unmodified) ardupilot source, essentially following simple suggestions from @gmorph in a more specific arduboat discussion thread about alternative heading input that I started quite a while back.

In summary, the essential steps for me, presently in a Windows XP OS, were:

  • Install Pixhawk Toolchain for Windows;
  • Run the PX4 Software Download (which does not fetch ardupilot per se);
  • In MINGW32 console window (“PX4 Console” of PX4 Toolchain), cd to my c:/pixhawk_toolchain/Firmware directory and do

git clone https://github.com/ArduPilot/ardupilot

  • Then cd to the ardupilot directory (my c:/pixhawk_toolchain/Firmware/ardupilot) and do

git submodule update --init --recursive

  • And then finally I was able to cd to the vehicle directory (my c:/pixhawk_toolchain/Firmware/ardupilot/APMrover2) and do

make px4-v2

…which resulted in the desired APMrover2-v2.px4 file in the same directory, which I successfully uploaded to the Pixhawk via Mission Planner (“Custom Firmware”) and ran OK.

There were some apparent errors reported during the build, for example

fatal: not a git repository: c:/pixhawk_toolchain/Firmware/ardupilot/modules/PX4nuttX/nuttx//…/…/.git/modules/modules/PX4NuttX

…which apparently did not preclude ultimately building the APMrover2-v2.px4 file. Incidentally, I copied the above carefully, and and don’t even understand the significance of the occurrence of “//”, nor of the “…/…/” …are we referring to a directory two levels back up at that point in the path??

The victory is a bit hollow for me since as yet I lack understanding of the “how” and “why” and “what else might” about this working, but needless to say I’m glad to keep going forward, with much gratitude to you all for help and insight.

@skyscraper, I think I kind of share your sense, albeit with less experience, about C++ in general and about Bjarne Stroustrup’s book in particular. (I have “The C++ Programming Language” 3rd edition, 1997.) I think he’s got insight to offer beyond just programming, and I enjoy the abstractions which I can relate to other areas of interest.

Best regards to all,
Pete

Perhaps to conclude this thread, this is meant to offer my own explicit answer to the question that I posed regarding the organization, in file hierarchical structure, of resources for building ardupilot code.

To recap, I was working in Windows environments, first XP and then Windows 7, and carried the process through to successful build of APMrover2 vehicle executable by the four steps:

  • Install pixhawk toolchain (obtained from http://firmware.us.ardupilot.org/Tools/PX4-tools/);
    This created “drivers”, “eclipse”, “Python27”, “toolchain”, and “tools” directories under C:/pixhawk_toolchain/. The tree at this point is like this.

  • Run “PX4 Software Download” part of pixhawk toolchain;
    This adds “Bootloader” and “Firmware” directories to c:/pixhawk_toolchain/. At this point c:/pixhawk_toolchain/Firmware contains directories “.git”, “cmake”, “Debug”, “Documentation”, “Images”, “integrationtests”, “launch”, “mavlink”, “misc”, “msg”, “NuttX”, “nuttx-configs”, “posix-configs”, “ROMFS”, “src”, “test_data”, “Tools” (not to be confused with “c:/pixhawk_toolchain/tools”), and “unittests”. It does not yet contain “ardupilot”. Except that hidden files and folders are not shown, the tree at this point is like this.

  • Start MINGW32 console window by the “PX4 Console Window” part of pixhawk toolchain, and after “cd Firmware”, do “git clone https://github.com/ArduPilot/ardupilot”;
    This adds the “ardupilot” directory to c:/pixhawk_toolchain/Firmware/. The seven module directories “gbenchmark”, “gtest”, “mavlink”, “PX4Firmware”, “PX4NuttX”, “uavcan” and “waf” are created under c:/pixhawk_toolchain/Firmware/ardupilot/modules/, but they are empty at this point. The tree at this point looks like this.

  • In MINGW32 console, after “cd ardupilot”, do "git submodule update --init --recursive;
    This fills the above-mentioned seven module directories, registering not only those seven modules but also Tools/gencpp, Tools/genmsg (the “Tools” directory for these two being c:/pixhawk_toolchain/Firmware/ardupilot/mk/PX4/Tools, not to be confused with either “tools” or “Tools” directories mentioned in step 2), mavlink/include/mavlink/v1.0 (that is, c:/pixhawk_toolchain/Firmware/mavlink/include/mavlink/v1.0, not down the c:/pixhawk_toolchain/Firmware/arupilot/modules/mavlink path), src/lib/DriverFramework, src/lib/ecl, src/lib/matrix, and src/modules/uavcan/libuavcan (all down the c:/pixhawk_toolchain/Firmware/src path). The tree at this point looks like this.

  • In MINGW32 console, after “cd APMrover2”, do “make px4-v2” (or initiate a build within eclipse);
    This produces the ~700kB “APMrover2-v2.px4” firmware file in the same vehicle directory.

I concede that it’s probably better not to concern oneself with such details, and that a snapshot account of resource organization like this is of dubious value in the ephemeral world of software development and repository management; but I thought it might be helpful by way of specific illustration, to one starting out, perhaps with the innocent expectation that code-building should be a fairly straight-forward two-step proposition of gathering files of source code into one place and then compiling it.

Cheers,
Pete

That step is not needed. That is actually downloading another flight controller software that you don’t need if you are using ArduPilot.

@OXINARF, Thank you very much for this clarification, Francisco. I just proved for myself that the “PX4 Software Download” step was extraneous; starting again from scratch, I successfully downloaded all necessary source from MINGW32 console by “git clone https://ArduPilot/ardupilot” followed by cd ardupilot and “git submodule update --init --recursive”, and then successfully built APMrover2-v2.px4 firmware by “make px4-v2” from the APMrover2 directory.

Cheers,
Pete

1 Like

Apologies if this question is too basic, but I think this thread that I started over a month ago got moved.

You all motivated me to get going with “development” using Linux, not just Windows; I finally got lubuntu 16.10 going on an old Thinkpad T-40. Anyway, attempting to build APMrover2 by make (following “Building Ardupilot for Pixhaw/PX4 on Linux Using Make”, even if make is being deprecated), there is this error, which I’m not adept enough to interpret.

If an expert eye can help me learn a bit, I’d once again be most grateful!

I suspect by invoking make by itself your trying to build the APM target which isn’t supported any more which is why its failing. You want to use waf. I’m going to assuming your building for a pixhawk. From the ArduPilot directory run

./modules/waf/waf-light configure --board px4-v2
./modules/waf/waf-light build --target bin/ardurover

That should work.

Thanks, Grant.

Thank you, Grant,

Actually, I should have said, from the ardupilot/APMrover2 directory, I invoked

make px4-v2

just as I had done when working on a Windows machine in a MinGW32 console a few weeks ago, to successfully build APMrover2.px4. But on the Linux machine it fails with the errors indicated by the link I gave.

Anyway, I tried the waf approach suggested and “configure” (invoked exactly as you suggested) produces the errors

Checking for header endian.h : not found
Checking for header byteswap.h : not found
Checking for program ‘cmake’ : not found
Could not find the program [‘cmake’]

Your missing some dependencies. Firstly make sure your repository is up to date (I assume its already setup and you have run “git submodule init”. From the ardupilot directory run:

git pull --rebase
git submodule update

Make sure you have read through all this

http://ardupilot.org/dev/docs/building-px4-for-linux-with-make.html#building-px4-for-linux-with-make

and you have run the pre-reqs installation script.

When using waf this has the instructions.

Thanks, Grant.