Servers by jDrones

GSoC 2019: AirSim Simulator Support for ArduPilot SITL

gsoc
gsoc19
(Rajat Singhal) #1

Introduction

AirSim​ is an open-source, cross-platform simulator for drones and cars, built on Unreal 3D Engine.
It provides​ physically and visually realistic simulations​ with external flight controllers such as PX4 or it’s own built-in controller simple_flight using either Software-In-The-Loop (SITL) or Hardware-In-The-Loop (HITL).​ It is generally used for testing software based on Computer Vision, Deep Learning, etc. & for generating large amounts of visual data which is essential for tasks such as Reinforcement Learning & Deep Learning.

During this summer as my GSoC 2019 project, I will endeavour to add active support for AirSim for ArduPilot SITL, specifically Copter platform. AirSim as a more advanced & realistic simulator (more resource hogging as well) can provide a basis for working on VIO algorithms & Obstacle Avoidance systems in Ardupilot within a simulation environment. I feel that this project will increase the adoption and usage of Ardupilot in cutting-edge research areas such as Computer Vision, Deep Learning & Reinforcement Learning for autonomous drones and vehicles, which are becoming increasingly popular these days.

The following is a list of features which I would like to add or support during this project (probably will be added in the same order chronologically) -

1. Support Normal SITL interface

This will involve initially adding support for the older Solo interface (see this for the Airsim side) in the Ardupilot SITL. Then modifying the AirSim code as well the previous SITL code to add lock-step scheduling for proper timing. Furthermore, the entire sensor packet layout will be changed to a JSON or XML-based one which would be easily extendable to add new fields, sensors, etc.

2. Manual Flying using RC

This would be smaller modification to add support for flying using a RC transmitter with a USB interface as seen in the video. Who wouldn’t like to fly in such beautiful & detailed environments available for Unreal Engine :slight_smile:

3. Add support for Lidar Sensor

AirSim has Lidar sensor with a large number of configurable settings which can be seen here.
Possibly even test the Copter Object Avoidance

4. API Support, ROS Integration

AirSim has it’s own Python & C++ APIs for interfacing with the drone as well as Image API for accessing the visual data being generated. Furthermore, there are also ROS packges for the same (see here & here)
Adding support for these APIs & ROS integration will enable developers to write and test their algorithms in simulation before running them on hardware.

Documentation

The documents and reports that will be created along the way will be added to the wiki, specifically:

  • Blog Posts on discuss.ardupilot.org for progress reports and to get feedback from the community
  • New page in the Wiki for the AirSim setup and usage with Ardupilot similar to the existing ones for the other simulators
    Sample programs, guides for testing things like the Copter Avoidance will also be added

Addditional

There are a few other things which I would like to add but which probably won’t be able to get finished in the summer. But these will definitely be worked on after the period ends in that case

1. Multi-Vehicle Simulation

AirSim allows multi-vehicle simulation and has ROS tutorials for the same

2. ArduRover Support in ArSim

AirSim has Car Vehicle as well but doesn’t have any external controller support till now. Adding support for using AirSim as a simulator for Rover will be great addition!!

3. Deep Learning or Computer Vision using Ardupilot & AirSim

Trying out Deep Learning using Ardupilot & AirSim, maybe blog posts for something like Monocular Depth Estimation, Optical Flow for estimating velocity using a downward-facing camera, so many possibilities!

Progress

I have opened a PR for the initial interface, there’s also a Readme with links for setting up AirSim and using with Ardupilot if anyone’s interested. The status of the project will be updated frequently on this blog as well as on any upcoming ones

6 Likes
(tridge) #2

thanks Rajat! good progress so far

(SATYABRATA SATAPATHY) #3

@rajat2004 we have developed a code snippet at Asteria to manually control drone from rc input / Logitech joystick.
Let’s see how to collaborate and take it forward.

(Rajat Singhal) #4

@satyasrmist Could you provide some more detail about the code which you’re talking about, I’m not exactly sure what you mean (is it a device driver, or something else?) and how it can be incorporated with this project

(SATYABRATA SATAPATHY) #5

The code has mavlink parser functions that accepts command from QGC and pass to Airsim API through a RPC client connection.

(Anthony Short) #6

This is a very interesting project - it would be great to have AirSim working well with ArduCopter. A couple of questions …

What would a good dev environment be? It looks like AirSim is more mature on Windows, but some of the other pieces like ROS might be better on Ubuntu. There are a lot of “moving parts” in this - are they all on one machine, or can they be on two machines (or one physical + 1 virtual machine)? Could you outline what you’d recommend for a good dev setup, or describe what you are using.

Also, do you have a sense of likely future progress on the AirSim project? They seem to have a lot of promised features (Unity, Ubuntu, etc), but the GitHub repo hasn’t seen much activity in the past six months, and their Trello board was last updated six months ago. Their Facebook group is very quiet. Is AirSim still being actively developed and supported?

(Rajat Singhal) #7

That could be very useful for creating a better interface for the users through QGC instead of just the API.
But I think that this would be more suitable to be integrated into the Airsim repo directly rather than with Ardupilot. Please correct me if I’m wrong

(Rajat Singhal) #8

@Anthony_Short Thanks for your interest in this project!!

I’m currently working on Ubuntu right now (haven’t used Windows for development before), I hope to get everything working on it first and then move on to Windows and try to fix any problems which occur. But yes, AirSim is more mature on Windows.
I think that for ArduPilot development, both would be similar only, especially with WSL (though I’m biased towards Linux). But for the AirSim side, Windows might be a better option

Right now, I’m working on a single machine with Ubuntu 16.04. However it should work on different machines as well (maybe 1 with Ardupilot & Ubuntu and another with Airsim running on Windows)
The current PR has a Readme with links for installation and also using it on different systems, I haven’t tried it out till now but will try to test it as soon as possible.
I’m not very sure that it would work with some part of it on a virtual machine, since there’s a lot of networking going on with Airsim connecting to Ardupilot or PX4 through sockets and the API also using RPC for connecting, one more thing to try out :slight_smile:

Edit: Just remembered about this - https://github.com/microsoft/AirSim/blob/master/docs/docker_ubuntu.md
You could try this out, but this is running everything inside the container

Sorry, don’t have much info regarding this. Their Trello board hasn’t been updated in quite some time, they have added several new features since then, especially in APIs and ROS packages & tutorials.
The development over there has slowed down over the last month, hopefully, it doesn’t last too long.

(SATYABRATA SATAPATHY) #9

You are right. We are doing further development in that way.