Integrating Depth Camera with ArduPilot Gazebo for Obstacle Avoidance

[Jazx_Jazx]

Hello community,

I am currently working on integrating a Realsense D435 with my drone in the ArduPilot Gazebo simulation environment. My goal is to implement obstacle avoidance during the drone’s mission. I have been using the ardupilot_gazebo package.
Maybe you could tell me how to combine all these things step by step. I will be extremely grateful
Here are the specifics of my setup:
Ubuntu 20.04 ROS kinetic
ArduPilot Gazebo version: GitHub - ArduPilot/ardupilot_gazebo: Plugins and models for vehicle simulation in Gazebo Sim with ArduPilot SITL controllers
Depth camera model: GitHub - issaiass/realsense_gazebo_plugin: || The realsense gazebo plugin for intel cameras from PAL Robotics || d435 I think i can use this : https://github.com/thien94/vision_to_mavros/blob/master/scripts/d4xx_to_mavlink.py
I could not successfully integrated the depth camera with the drone in ArduPilot Gazebo . I would appreciate any guidance or insights on how to proceed with this task.

Specifically, I am interested in:

How to configure ArduPilot to utilize the depth camera data for obstacle detection.
Are there any existing plugins or tools that facilitate this process?
Any code snippets or examples demonstrating obstacle avoidance using depth cameras with ArduPilot in Gazebo.
I have already reviewed the relevant documentation, but additional insights from the community would be immensely helpful. Thank you in advance for your assistance!

[rhys (Rhys Mainwaring)]

Hi @Jazx_Jazx - a full answer to your question will probably take a few posts, but the first part asking to integrate a depth camera into your simulated drone should not be too difficult.

Gazebo Garden and Harmonic support depth cameras directly, so you may not need to used the one from the plugin (you may need to tune the camera intrinsics to match, which would be for a follow up).

I’ve added a branch to ardupilot_gazebo here: GitHub - srmainwaring/ardupilot_gazebo-1 at prs/pr-all-camera-sensors that adds all the available camera sensors to the gimbal mount.

The example is run using

gz sim -v4 -r gimbal.sdf

You can inspect the behaviour of each sensor by selecting the appropriate topcis in the Image Display.

Figure: depth camera

Figure: rgbd camera

Figure: thermal camera

Figure: wide angle camera

Figure: panoptic segmentation camera

The next step is to integrate the data from the camera topics with ArduPilot. For that we’ll probably want to use the tools available in ROS to carry out any image manipulation required, or at least convert the image data into a format recognised by ROS which will be more accessible to other tools. For this we’ll use the ros_gz bridge - more on that in the next post.

[Jazx_Jazx]

Thank you very much for such a detailed guide!!!