When was the last time someone showed you a camera capable of object detection like this, without using a companion computer or any other external hardware? Today I’d like to introduce a camera capable of this, and much more!
For the past few months, I have had the opportunity to work on a fascinating new Camera. It is the OAK-D series from Luxonis. I have been experimenting with the original OAK-D and the OAK-D-IOT-75 for a while now. In this blog (Part 1 of a series of blogs I hope to post), I want to give a small description of them:
OAK-D:
This is essentially an RGB-D Camera (Stereo Camera + RGB Camera system, much like the Intel RealSense D435i). If you are not familiar with Stereo Cameras, they help us essentially generate a wide 3-D point cloud (3-D geometrical coordinates to the image on a real-world scale) via two or more cameras placed parallel to each other. This can be used for several applications, like Obstacle Avoidance. This technology isn’t new, and we have been working with the RealSense cameras for a long time.
What’s new about this particular camera is its ability to run AI/Computer Vision algorithms onboard the camera. As you may be aware, most applications in Computer Vision and Drones require a heavy (potentially expensive) Companion Computer onboard. This is a considerable bottleneck for any user to experience this technology. Well, not anymore! Things like detecting humans, everyday objects, animals can be done at a relatively high refresh rate inside the camera (with no additional hardware required, just any low powered USB-enabled companion computer, like an RPi Zero). Even training the camera to detect custom objects for your tailor-made applications are super easy!
What makes this sensor even better for ArduPilot is the fact that the detected objects can be tracked in 3D space via the stereo camera. So in essence, the camera outputs the detected object and its 3D coordinates to the Host (The device where the camera is connected too). Alternatives like the RealSense series had to transfer the entire Image frames onto the host, which required USB3 (Terrible for EM noise!) and a relatively fast Companion Computer. Other neat features include: It has onboard scripting capabilities (much like Lua scripting for ArduPilot), inbuilt IMU, object tracker support via inbuilt EKF and other algorithms, amongst many many more features.
I would also like to mention some of the issues that I have faced with the camera: There are issues with focusing on the RGB camera when it is put in high-vibration environments. This is being addressed by Luxonis by launching a “fixed-focus” version of the camera in the near future. A comparison of fixed focus vs auto focus can be seen here:
The other issue is that the quality of the Depth Frames isn’t as good as the RealSense cameras. They definitely have more noise. I have found workarounds, though. The Luxonis team plans to launch OAK-D-PRO, which should again improve this.
OAK-D-IOT-75:
This is the EXACT same camera as the OAK-D. Except: it has an ESP32 (with WiFi and BT) on board, which has a perfect use case for us ArduPilot enthusiasts! The camera can directly talk to the ESP32, and the ESP32 can be used to interface with the Flight Controller (via the serial port). This means we can finally have Computer Vision/AI applications without needing any Companion Computer! Directly plug in the camera to your Flight Controller and no need for any other equipment!
This is the camera that I have spent the last few weeks integrating and making new example scripts for. If you are interested in my development and integration work, please read the second part of this blog for some interesting applications!