The 2020 GSoC ArduPilot summer project has officially started. This year, I will have the privilege of working with @ppoirier and @jmachuca77. Thank you for letting me having another great summer with ArduPilot and I hope we will have a fun and productive time working together!
1. Introduction
Unmanned Aerial Vehicle (UAV), commonly known as drone, is becoming an essential tool for a variety of industries such as mapping, delivery, surveillance, warehouse, and agriculture, to name but a few. However, their potential is currently mostly limited to outdoor scenarios because their flight capability relies on GPS signal, which becomes unstable or unavailable in many other situations such as indoor, under a bridge, between tall buildings, in a tunnel, etc. Not only that, another crucial component for fully autonomous operation, which is the ability to avoid obstacles that could pop up unpredictably during the mission, is often only available in high-end proprietary products that cost thousands of dollars, or too complex of a solution to be applicable for the sensor and processing power onboard low-end drones.
Navigation in non-GPS environments while avoiding obstacles, therefore, plays a critical role and is a highly desired feature for any autonomous robot. VIO tracking camera and depth camera when appropriately combined can offer both non-GPS navigation and obstacle-sensing abilities that don’t overexert the processors of low-end drones and, perhaps more importantly, keep plenty of room for further development by the user.
In this project, we will integrate and improve the usage of these technologies with ArduPilot, in an attempt to bring non-GPS navigation and obstacle avoidance into basic abilities of ArduPilot that everyone can rely on and build upon.
2. New enhancements coming to ArduPilot
The Intel Realsense T265 has been established as a “plug-and-play” sensor for ArduPilot from this 2019 GSoC project. Detailed introductions can be found on the following wiki pages:
Object avoidance with rotating LiDAR has also been long available. However, there remains room for improvements. This project would contribute the below benefits to the community, which are also our main objectives:
Enhancements of the vision-related codebase
Summary: The handling of vision-related messages in the ArduPilot codebase has been moved under the AP_VisualOdom library. Certain improvements that can be introduced during this project include:
- Enhancement of the handling of VISION_POSITION_ESTIMATE and VISION_POSITION_DELTA messages with the Intel Realsense T265 for both non-ROS and ROS.
- Detect and pass the SLAM’s tracking position reset counter from the VIO SLAM on the T265 to AP using “reset_counter” MAVLink2 extension field.
- Another major improvement can be done by measuring the lag between camera and autopilot’s IMU and ensuring that the EKF is using the correct lag.
- Send velocity information to AP, either by adding support for VISION_SPEED_ESTIMATE or ODOMETRY, which can carry all the information provide by the Realsense T265 into one single message.
Integration of Realsense Depth camera
Summary: Develop method(s) to convert the depth image into OBSTACLE_DISTANCE MAVLink messages, which is the straightforward way to interface with AC_Avoidance library.
- As of now, AP supports providing obstacle information in one of two types of messages: DISTANCE_SENSOR and OBSTACLE_DISTANCE. Among the two, OBSTACLE_DISTANCE is the better choice for it provides an array of distances, which should better represent the depth data. Hence, data from depth image will be converted to OBSTACLE_DISTANCE message, which is then sent to the FCU.
- First, it should be made clear how the obstacle in the environment is represented by the OBSTACLE_DISTANCE message as well as by the depth image. Once the underlying principles are clear, a conversion can be made from the depth image to the messages. The messages can then be sent to the FCU at a predefined frequency.
- We can now follow the OBSTACLE AVOIDANCE wiki page to perform testing. Most importantly, trials should be carried out to establish good default values for parameters and the system can then be released for beta testers (blog posts).
- Once the performance is verified by a number of testers, official documentation can be made (wiki page).
Integration of Tracking + Depth cameras (SLAM+ROS)
Summary: while tracking and depth cameras can be used separately with ArduPilot, this objective looks for a SLAM-based approach leveraging the Robot Operating System (ROS) framework to combine both cameras into one package to extend the potential use cases of ArduPilot.
- To combine the two cameras together, first we need to calibrate the extrinsics between cameras (with any types of mounting and not just the provided one) based on OpenCV and Python.
- Synching the data streams between the cameras is also important. Here’s a working example.
- There are several SLAM examples such as this occupancy-mapping to generate 2D occupancy map, but RTABMap or Google Cartographer are also plausible alternatives.
- The next step is to convert the data from the two cameras into the format that the new SLAM package accepts and into the position + obstacle data (with support added above) to be sent to AP.
To properly process the all the data from the two cameras (depth image, pose, potentially fisheye images), a new companion computer class will be introduced to ArduPilot, namely the Up Squared.
GPS to non-GPS transition
Summary: Seamless transitition between GPS and non-GPS environment is a highly desirable feature. Several solutions are currently under development and initial results are promising.
- A direct approach, which are being actively developed and tested by @rmackay9, @anbello and @ppoirier, is using the
VISION_POSITION_DELTAand EKF3 (which will also have lots of development coming) to use the delta in position and rotation data to assist the transition. - Another potential solution is to use additional sensor information to detect if the vehicle is indoor or outdoor. For example, a sonar/1D range sensor can be mounted upward (point to the ceiling) to detect the transition from indoor (range data available) to outdoor (no range data) and perform indoor-outdoor transition.
- One can also think of a solution, when the camera can see at least part of the sky, to use images to detect when the vehicle is out in the open and when it is not, to facilitate the transition between GPS and non-GPS localization data. This has been done before in research
3. Useful References
- Easier setup for Intel Realsense T265 by @rmackay9.
- Using the Realsense T265 with MAVCONN library (cpp, non-ROS) to send various kinds of messages to AP by @anbello: blog post and Github repo.
- Experiment with Visual Odometry - ROVIO part 1 and part 2 by @ppoirier.
- Indoor flight with external navigation data by @chobitsfan.
- 2019 GSoC Project (integration of the tracking camera Realsense T265 and ArduPilot).
4. Progress Update
The realization of the aforementioned enhancement objectives would require clear and detailed system requirements, implementation plan and deliverables, which will be gradually published in the form of blog posts. The posts are also there to receive feedback/suggestions and improve/modify the system as needed before moving on to the next step as well as support pages for users in the future.
As these blog posts are introduced, the list will be updated below this section for easy lookup. As usual, any input is welcome and I look forward to hearing from you all.
But at least you can get Octomap from RTAB easly. Definitely go for RTAB, e.g. you can compile and use directly for odometry VIN-Fusion and LOAM. Good luck, I will be following