Object Avoidance with ZED 3D Camera

Blog
, , , ,
1

Here are some videos of a successful test of ArduPilot (Copter)'s object avoidance using a forward facing StereoLabs ZED 3D Camera. The first video is the 3rd person view of the vehicle.

The vehicle is my now familiar EnRoute EX700 with a Pixhawk1 running Copter-3.5-dev paired with an NVidia TX1 and the ZED camera. On the TX1 was OpenKai which is a fully open source vision application being developed by Kai. OpenKai uses the ZED SDK to pull the pixel-level depth information from the camera. It then converts this into a DISTANCE_SENSOR mavlink message and sends it into ArduPilot.

This 2nd video shows the onboard video

You’ll likely notice from watching the onboard video, that the ZED camera is aimed up 20 degrees or so. The reason for this is that generally a multicopter leans forward in flight so it’s more important to have depth data from above than below. Also OpenKai does not yet adjust which part of the image it uses depending upon the vehicle’s attitude. This means that when flying at low altitudes (like in this test) it sees the ground and sends unnecessarily short distances to ArduPilot. This should be resolved in a future revision.

I’ve written up quite a detailed explanation of ArduPilot’s object avoidance on our developer wiki. This includes details on the algorithms we’re using and also on how sensor developers can get their proximity sensor data into ardupilot so that it can be used for object avoidance. I hope this helps bring others developers up to speed on this important feature that’s under very active development.

There are still many improvements we can make to the object avoidance to make it more robust and accessible to more users (see the issues list at the bottom of the wiki page linked) but it’s coming along well.

3 Likes
2

Awesome! Maybe using an aux servo to allow cam tilt (perhaps using _angle_bf_output_deg.y -= degrees(_frontend._ahrs.pitch) from the Mount libraries) would make the camera reading indipendent from vehicle attitude?

3

@Ferruccio,
Sorry for the slow reply…
Yes a servo is certainly an option. I like having less moving parts myself so I’ll be pursuing the “digital” solution but for sensors where we don’t get any vertical range (like lidar) a servo is a decent idea.

1 Like
4

@rmackay9, Hi, I need your guidance.
I am looking forward to implement obstacle avoidance in pixhawk cube running arducopter using zed-tx2-openkai. I would like to inquire the following:

  1. Did you find any delay in distance measurement using ZED camera with jetson TX1 running open KAI. This is very important in case of high speed copters i.e. 10-12 m/sec as if the distance measurement is late, it is very likely that the copter strikes to the object before it is detected and braking is applied.
  2. What is the effect of day light in distance measurements. In case of low sunlight, does the range or distance measurement is affected?

Thankyou
Regards
Yasir Khizar