GSoC 2018: Complex Autonomous Tasks Onboard a UAV using a Monocular Camera (NVIDIA Redtail)

In our previous blog post we discussed the hardware design and started porting the original Redtail project to support ArduPilot. Since we have upgraded the control method for Redtail controller and the simulation stack for trail following.

Redtail controller:

In the original Redtail project, px4’s waypoint navigation ability was used for navigation: in each frame, TrailNet CNN selects a new rotation angle based on the current heading of drone, and they would select a new point meters away from the drone toward the desired direction as its navigation position setpoint target. By relying on the drone’s ability to autonomously navigate to that point they would move to the desired direction. By keeping updating the setpoint 20 times per second they achieved a smooth trajectory on trails.

Using the exact same method was very slow on ArduCopter, no matter how far from the drone the target was selected. It also relies heavily on odometry and position finding in a GPS denied environment using optical flow and rangefinder.

In addition to porting this method to work on ArduCopter, we also implemented a linear velocity tracking method, which is easily scalable to different speeds, more reliable, and based on our test the drone can follow trails more accurately.


By @khancyr’s suggestion, we have used Google maps API to load satellite images of trails as the ground plane of Gazebo simulator.
We have also modified the Gazebo IRIS model so that it’s gimbal camera is fixed in 0.6 rad and uses Gazebo ROS camera plugin to publish its video stream over ROS topic.
Although it’s in 2D and the ground plane quality is very low, with some modification we were able to get the redtail project working on this simulator using TrailNet CNN model and autonomously navigate through tails on the map.
For more information about how to install and use this software stack please refer to this tutorial


Contributions during Google Summer of Code:

All the changes on this project during GSoC can be found from this Github comparison link

Projects state:

We started some different but related subprojects during this GSoC period, some of which need a little more time to conclude. Although the GSoC period is coming to an end, I am planning to finish all the following works:

  • [Done] Design and build the drone
  • [Done] ArduCopter SITL + Gazebo + Google maps
  • [Done] ArduCopter support for Redtail project (tested on simulator, some debug needed on the drone)
  • [Needs more time] Using the hardware for more monocular camera autonomous tasks (most software stacks and CNNs for our previous human-UAV interaction works are ported to NVIDIA Jetson and ArduCopter, but needs to be finished and tested on the hardware)
  • [Needs more time] Making the latest PX4Flow firmware compatible with ArduPilot (some of the source code were adopted to increase the brightness and reliability in low light environments, but making the firmware fully compatible is yet to be done)


I have to thank @jmachuca77, @rmackay9, and @khancyr again for their time, helps and patience.