GSoC 2019: AirSim Simulator Support for ArduPilot SITL

Introduction

AirSim​ is an open-source, cross-platform simulator for drones and cars, built on Unreal 3D Engine.
It provides​ physically and visually realistic simulations​ with external flight controllers such as PX4 or it’s own built-in controller simple_flight using either Software-In-The-Loop (SITL) or Hardware-In-The-Loop (HITL).​ It is generally used for testing software based on Computer Vision, Deep Learning, etc. & for generating large amounts of visual data which is essential for tasks such as Reinforcement Learning & Deep Learning.

During this summer as my GSoC 2019 project, I will endeavour to add active support for AirSim for ArduPilot SITL, specifically Copter platform. AirSim as a more advanced & realistic simulator (more resource hogging as well) can provide a basis for working on VIO algorithms & Obstacle Avoidance systems in Ardupilot within a simulation environment. I feel that this project will increase the adoption and usage of Ardupilot in cutting-edge research areas such as Computer Vision, Deep Learning & Reinforcement Learning for autonomous drones and vehicles, which are becoming increasingly popular these days.

The following is a list of features which I would like to add or support during this project (probably will be added in the same order chronologically) -

1. Support Normal SITL interface

This will involve initially adding support for the older Solo interface (see this for the Airsim side) in the Ardupilot SITL. Then modifying the AirSim code as well the previous SITL code to add lock-step scheduling for proper timing. Furthermore, the entire sensor packet layout will be changed to a JSON or XML-based one which would be easily extendable to add new fields, sensors, etc.

2. Manual Flying using RC

This would be smaller modification to add support for flying using a RC transmitter with a USB interface as seen in the video. Who wouldn’t like to fly in such beautiful & detailed environments available for Unreal Engine :slight_smile:

3. Add support for Lidar Sensor

AirSim has Lidar sensor with a large number of configurable settings which can be seen here.
Possibly even test the Copter Object Avoidance

4. API Support, ROS Integration

AirSim has it’s own Python & C++ APIs for interfacing with the drone as well as Image API for accessing the visual data being generated. Furthermore, there are also ROS packges for the same (see here & here)
Adding support for these APIs & ROS integration will enable developers to write and test their algorithms in simulation before running them on hardware.

Documentation

The documents and reports that will be created along the way will be added to the wiki, specifically:

  • Blog Posts on discuss.ardupilot.org for progress reports and to get feedback from the community
  • New page in the Wiki for the AirSim setup and usage with Ardupilot similar to the existing ones for the other simulators
    Sample programs, guides for testing things like the Copter Avoidance will also be added

Addditional

There are a few other things which I would like to add but which probably won’t be able to get finished in the summer. But these will definitely be worked on after the period ends in that case

1. Multi-Vehicle Simulation

AirSim allows multi-vehicle simulation and has ROS tutorials for the same

2. ArduRover Support in ArSim

AirSim has Car Vehicle as well but doesn’t have any external controller support till now. Adding support for using AirSim as a simulator for Rover will be great addition!!

3. Deep Learning or Computer Vision using Ardupilot & AirSim

Trying out Deep Learning using Ardupilot & AirSim, maybe blog posts for something like Monocular Depth Estimation, Optical Flow for estimating velocity using a downward-facing camera, so many possibilities!

Progress

I have opened a PR for the initial interface, there’s also a Readme with links for setting up AirSim and using with Ardupilot if anyone’s interested. The status of the project will be updated frequently on this blog as well as on any upcoming ones

8 Likes

thanks Rajat! good progress so far

@rajat2004 we have developed a code snippet at Asteria to manually control drone from rc input / Logitech joystick.
Let’s see how to collaborate and take it forward.

@satyasrmist Could you provide some more detail about the code which you’re talking about, I’m not exactly sure what you mean (is it a device driver, or something else?) and how it can be incorporated with this project

The code has mavlink parser functions that accepts command from QGC and pass to Airsim API through a RPC client connection.

This is a very interesting project - it would be great to have AirSim working well with ArduCopter. A couple of questions …

What would a good dev environment be? It looks like AirSim is more mature on Windows, but some of the other pieces like ROS might be better on Ubuntu. There are a lot of “moving parts” in this - are they all on one machine, or can they be on two machines (or one physical + 1 virtual machine)? Could you outline what you’d recommend for a good dev setup, or describe what you are using.

Also, do you have a sense of likely future progress on the AirSim project? They seem to have a lot of promised features (Unity, Ubuntu, etc), but the GitHub repo hasn’t seen much activity in the past six months, and their Trello board was last updated six months ago. Their Facebook group is very quiet. Is AirSim still being actively developed and supported?

That could be very useful for creating a better interface for the users through QGC instead of just the API.
But I think that this would be more suitable to be integrated into the Airsim repo directly rather than with Ardupilot. Please correct me if I’m wrong

@Anthony_Short Thanks for your interest in this project!!

I’m currently working on Ubuntu right now (haven’t used Windows for development before), I hope to get everything working on it first and then move on to Windows and try to fix any problems which occur. But yes, AirSim is more mature on Windows.
I think that for ArduPilot development, both would be similar only, especially with WSL (though I’m biased towards Linux). But for the AirSim side, Windows might be a better option

Right now, I’m working on a single machine with Ubuntu 16.04. However it should work on different machines as well (maybe 1 with Ardupilot & Ubuntu and another with Airsim running on Windows)
The current PR has a Readme with links for installation and also using it on different systems, I haven’t tried it out till now but will try to test it as soon as possible.
I’m not very sure that it would work with some part of it on a virtual machine, since there’s a lot of networking going on with Airsim connecting to Ardupilot or PX4 through sockets and the API also using RPC for connecting, one more thing to try out :slight_smile:

Edit: Just remembered about this - AirSim/docs/docker_ubuntu.md at main · microsoft/AirSim · GitHub
You could try this out, but this is running everything inside the container

Sorry, don’t have much info regarding this. Their Trello board hasn’t been updated in quite some time, they have added several new features since then, especially in APIs and ROS packages & tutorials.
The development over there has slowed down over the last month, hopefully, it doesn’t last too long.

You are right. We are doing further development in that way.

@rajat2004 can you tell which parameters you used to fly the quad. Whenever I connect AirSim to Arducopter SITL, after arming the quad is very unstable and doesn’t fly.

@akshatpandya Hi, Thanks for your interest and for trying it out!
I have added the parameters file in the PR, here’s the link from my tree - https://github.com/rajat2004/ardupilot/blob/sitl-airsim/libraries/SITL/examples/Airsim/quadX.parm

You can add it using the -add-param-file option as following -
sim_vehicle.py -v ArduCopter -f airsim-copter --add-param-file=libraries/SITL/examples/Airsim/quadX.parm --console --map

Please let me know if the performance is better with this and if there are any other problems which you face, feedback is greatly appreciated!

Sorry, forgot to mention
Please try this change on the AirSim side, with the wall-clock time, the firmware doesn’t remain in sync with the physics engine

Got MAVLink msg: COMMAND_ACK {command : 400, result : 4} APM: PreArm: EKF attitude is bad
APM: PreArm: Compass not healthy
APM: PreArm: Need 3D Fix

I’m getting this response when I try to arm in loiter mode. Even in guided mode I see same thing. The compass variance is too high. Can it be that the sensors values are not being received/sent properly?

Also on connecting to mission planner, it doesn’t get location and hence unable to switch to modes that use GPS.

Am I doing something wrong? @rajat2004 Can try it once by connecting mission planner?

Really sorry for the late reply, I had been quite sick for the past few days so this got missed.
One thing is that the arming can take a few seconds, try arming a few times and then wait, generally, it disappears after a while. I haven’t tried with Mission Planner (I use Mavproxy only :slight_smile: ) ,will install and let you know if I have the same problem.

Also, please try and run the updated PR with the Airsim branch mentioned here - https://github.com/ArduPilot/ardupilot/pull/11367#issuecomment-507602931

Hope this helps!

Been quite some time since I have updated here

There have been several developments till now, first is that the initial PR has been merged into ArduPilot master!

AirSim PR is open

Follow-up PR for adding Lidar is also up!

Finally, PR for adding a Wiki page for AirSim setup and usage is also there

The entire setup has been tested on Ubuntu 16.04 as well as Windows 10 with Ardupilot running in WSL Ubuntu 18.04.
Instructions for using custom environments are also present in the Wiki PR

Do try out the new additions and let me know if there are any problems!