GSoC 2019: AirSim Simulator Support for ArduPilot SITL Part II

Hello!
This is the second post on my GSoC project of integrating AirSim simulator with ArduPilot SITL.
This will also be my submission for the GSoC final evaluation, where links to PRs and code is needed. So the post will have links to my PRs at some places, do excuse me for the same.

In my previous post, I had written about AirSim and the features I would aim to develop over the course of the project. All of the things mentioned henceforth have been merged into ArduPilot & AirSim (unless mentioned otherwise) and documentation for the setup and usage can be found at - http://ardupilot.org/dev/docs/sitl-with-airsim.html

1. Copter SITL with AirSim

The first thing which was done was to build the interface from both ArduPilot and AirSim’s side for the basic SITL usage. I haven’t personally tested all the flight modes but the common ones such as Stabilize, Guided, Loiter, PosHold, Auto, RTL, Circle, etc. have been tested to be working.

A few key points about the interface -

  • Lock-Step Scheduling - This is where the Physics runs with a constant timestep, with each step being synchronized with the firmware control. This allows attaching a debugger to either the SITL or even AirSim, stop at breakpoints, resume, all without any effect on the physics.

  • JSON packets - The sensor data from the simulator to ArduPilot is sent as JSON packets. This allows easy addition or removal of sensors and fields in the packet as needed without breaking the existing interface. This has been used for adding Lidar and RC Control.

  • Cross-Platform - The setup has been tested to be working on Linux (Ubuntu 16.04) as well as with AirSim on Windows & SITL in WSL Ubuntu 18.04

  • Running on different machines - AirSim being a more resource-consuming simulator, it can be a common usage to have it run on a separate system than ArduPilot or even different OSes. The ports can also be changed as needed

PR in ArduPilot - https://github.com/ArduPilot/ardupilot/pull/11367 (Merged)
PR in AirSim - https://github.com/microsoft/AirSim/pull/2075 (Single PR, Merged!)

2. Lidar

AirSim has support for Lidar Sensor and adding this was essential for using AirSim as a suitable simulator for working on Obstacle Avoidance and related tasks. Usage instructions are in Wiki.

Note: Very recently, Dijkstra & BendyRuler Avoidance was added in Copter 3.7.0 Dev, haven’t tested it again with these but will do so soon and make changes if needed

PR in ArduPilot - https://github.com/ArduPilot/ardupilot/pull/11835 (Merged, has RC and Multi-Vehicle as well)

3. Manual Flying using RC

RC can be used with AirSim, and the same data is passed to SITL, now you can enjoy flying as crazily as you want in these nice, scenic environments available in Unreal Engine!

4. Multi-Vehicle Simulation

Multi-vehicle simulation is another feature which AirSim has, and this works with ArduPilot as well!

An example script is present in libraries/SITL example folder and usage instructions are in the Wiki.
This script spawns 2 copters, with Follow mode enabled in the second one. Both can be controlled from any GCS such as Mavproxy or QGC, and have the second follow the first.

5. AirSim APIs

AirSim has a rich set of Python & C++ APIs for getting information and for controlling the movement of the vehicle.
After some discussion with my mentors, we decided that adding support for the movement APIs isn’t a priority for now since it anyways won’t be used on a real vehicle and there are other alternatives such as DroneKit.
The other APIs not related to the vehicle movement such as the Image APIs work correctly. There are some tutorials on using ROS with AirSim, please check airsim_tutorial_pkgs & airsim_ros_pkgs on using them.

Any method of controlling the movement which connects directly to ArduPilot rather than using AirSim’s API work, examples include DroneKit & ROS with Mavros.

Using Custom Environments

AirSim by default comes with a simple Blocks environment, which is a very lightweight world suited for development. The environment used in the video above is called Landscape Mountains, and there are many such detailed environments available in the Unreal MarketPlace which can be downloaded and used.

This interface would also be present in the next Release of AirSim, which has the AirSim plugin packaged with many different environments, for Windows and Linux, which will allow users to test it out much more easily without going through the entire setup.

Rover Vehicle Coming Soon!

After the work on Copter was done apart from fixes now and then, I started looking at adding support for Rover vehicle and experimenting with it. There are still some problems left to be figured out to get it working, but still, quite a lot of the work has been done!

Concluding

These past three months with ArduPilot have been a great time for me, from learning new things to being a part of a community, the experience was unparalleled. Especially since it was my first dive-down into working with large codebases, let alone two of them!
@tridge and @peterbarker have been incredibly helpful and considerate as mentors, really could not have asked for better! A huge shoutout to the other developers of ArduPilot, always there to help out when stuck, some of the early users who started using this and gave valuable feedback and lastly the developers of AirSim who gave advice on going about the project & later on reviewed and merged the code.
Though officially my GSoC work with Ardupilot is over, I look forward to continuing working with this team and taking the AirSim integration even further.

Finally, one last line with links to all the PRs for GSoC in ArduPilot main repo, ArduPilot Wiki and AirSim

9 Likes

Great post! It has been a real pleasure working with you on this GSoC project, and the result is a fantastic addition to the ArduPilot project. Many thanks for your hard work and persistence to see this through to such a good conclusion!
Cheers, Tridge

2 Likes

Are there any tricks to getting AirSim to talk to WSL2? I’m getting ‘No sensor message received - Success’

It seems to be connecting initially because home is set to the “OriginGeopoint” from the AirSim settings file.

I haven’t tested till now on WSL2, googling seems to indicate that there’s some problem with UDP specifically
This comment seems promising - https://github.com/microsoft/WSL/issues/4212#issuecomment-507772373
That along with https://ardupilot.org/dev/docs/sitl-with-airsim.html#run-on-different-machines might work, will post regarding any update after testing

1 Like

Yeah, so this does work for me. Steps I followed -

  1. Add UE4Editor.exe to an allowed rule, see https://github.com/mahmoudajawad/wsl-hacks/blob/master/wsl2-networking.md
    The .exe file can be found in Engine/Binaries/Win64 in the folder where UE has been downloaded
    If using a pre-built environment, add the corresponding .exe file in the rule

  2. Added the following in my .profile

export WSL_HOST_IP=$(ipconfig.exe | awk '/WSL/ {getline; getline; getline; getline; print substr($14, 1, length($14)-1)}')
export DISPLAY=$WSL_HOST_IP:0
  1. In settings.json, set the IP addresses-
"LocalHostIp": "0.0.0.0",
"UdpIp": "192.168.179.223",

UdpIp is the IP address of Ubuntu 18.04 running under WSL, using ip addr show eth0

  1. I’m using XLaunch for the X11 windows, in that I added -ac as additonal argument, see https://github.com/microsoft/WSL/issues/4106

  2. Launching SITL - sim_vehicle.py -v ArduCopter -f airsim-copter -A --sim-address=$WSL_HOST_IP

So many things are needed since the address spaces are different in WSL 2 rather than the same on WSL 1. Should add to the Wiki as well, after some more testing or checking if steps can be reduced

2 Likes

Works great! thank you!

hum, a little more testing and I have found the graphics are quite slow AP is reporting a constant 333Hz but the AirSim graphics seem to be running at less than 10 fps.

The framerate shown by AP is the Physics rate, the graphics rate doesn’t affect the physics in Multirotor
Have you disabled Use Less CPU when in background?

You could also try using the new releases, they generally run faster than the Editor

I’m using the blocks pre-built, I don’t think we get any options there? I’ll try with the local build now.

I would definitely be very surprised if the binaries have the CPU problem
If you’re using Lidar, Note that right now there seems to be a problem with the lidar visualization in AirSim, enabling DrawDebugPoints in that will reduce your FPS like anything and might crash due to excessive memory usage. This is set in the Lidar settings in the AP documentation, so setting DrawDebug in the Lidar sensor settings to false will be enough
Just got to know about this problem yesterday

Yeah seems to be lidar, getting 200 fps on blocks with no lidar down to about 3 with. I will have a play with the settings. Also got the same with local build.

Good to know that the problem was being caused by lidar. I’ve added that in the Wiki PR
This is a change in behavior than UE 4.18 where it used to work without any problems

Yep with DrawDebugPoints false I get 190 fps. Must be a AirSim bug/inefficiency there.

Flying around a little seems to show the left and right side of the lidar is swapped. I guess there is a clockwise vs anti clockwise issue somewhere.

I’m looking forward to using this properly, its certainly a very powerful tool, Thanks for the help!

Hi,

I am a young student trying to build UE mit wsl2 - Ubuntu 18.04 where I installed ardupilot for SITL and they most of the time run themselves. Added the rule to my windows inbound rules. Don’t know any .profile-data. What is and where is my “.profile” ?
Putting your two lines of code in my Powershell just throws errors. Is WSL_HOST_IP a placeholder which I can replace with localhost?

ONe more question: Where is settings.json ? It is not in the win64 nor in windows taskbar search finable

Hi,
Sorry for the slightly incomplete steps. I seem to be unable to edit that post now, so will describe the steps here, will add to the Wiki directly as well after testing on a fresh WSL install

  1. The first step mostly isn’t directly required, for e.g. when launching a binary, the Windows Firewall will mostly popup saying the .exe file has been blocked, in that enable it to access Private networks as well. You can still follow the steps in https://github.com/mahmoudajawad/wsl-hacks/blob/master/wsl2-networking.md and it should work, if someone could confirm if it’s required or not, that would be great!

  2. The .profile file is located in the home directory in the Ubuntu system in WSL2. Add the below lines (you can edit using vim, nano or some other editor) -

export WSL_HOST_IP=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}')
export DISPLAY=$WSL_HOST_IP:0 
  1. The settings.json is in the Documents/AirSim folder in Windows. It’ll be created by AirSim on startup if not found. The entire settings is mentioned in the Wiki. Modify the specific lines below -
"LocalHostIp": "0.0.0.0",
"UdpIp": "172.25.239.255",

UdpIp can be found by running ip addr show eth0 inside Ubuntu in WSL

  1. Set the XWindows application settings, in particular Disable Access Control. This is described in Wiki’s WSL SITL Setup page also. The Step 2 export is also similar, just that an extra WSL_HOST_IP variable is added since it’s used next.

  2. Launch SITL finally - sim_vehicle.py -v ArduCopter -f airsim-copter -A --sim-address=$WSL_HOST_IP

Do comment if something doesn’t work!

Thank you for your detailed instructions, and it worked for me! Below is some more feedback information from my tests, and I hope they might be helpful for others.

Test environment:
Windows 11 Pro build 22000.708, running Ubuntu 22.04LTS under WSL2. Installed Unreal Engine v4.27.2 via Epic Games Launcher. Complied AirSim under Visual Studio 2022 (Community) , and ArduPilot in WSL2 with waf.

Now, for Step 1. I’ve tested that you don’t have to follow the exact steps in https://github.com/mahmoudajawad/wsl-hacks/blob/master/wsl2-networking.md . Just check the boxes in your Windows Firewall to grant communications for UE4Editor.exe on Public and Private networks, and there’s no need to manually add new Inbound Rules.

For my case, Ubuntu 22.04LTS in WSL2 comes natively with graphical support via WSLg, so I didn’t add export DISPLAY=$WSL_HOST_IP:0 to .profile, nor the command described in WSL Setup Steps. I also didn’t install any additional XWindows app.

Other steps are done exactly as what you mentioned in your comment, and simple_vehicle.py example works perfectly fine, following the official Wiki Using SITL with AirSim.

I will be running more tests and maybe come back to this thread. Thanks again!