Hi, one of my old projects is a real time stereo camera system running entirely on an FPGA. The purpose is to produce super low latency depth maps, which it does, but I haven’t integrated it into a larger system yet, so it just sits on my desk… I’m thinking I could integrate it into arudpilot, in a local build, and use it for low-level long-distance autonomous (plane) flights, aided by obstacle avoidance. I’m not sure what strategy I would use though.
To keep it simple, I could simply output a high signal indicating that the aircraft needs to pull up.
I could have the stereo system out of the loop, and have it override the ardupilot’s servo output for elevator. That would likely cause the ardupilot system to get confused though…
Thanks. I read the article. I need more info though.
I need to see example serial data
I need a summary of what parameters need to be changed so that ardupilot will recognize and use this data correctly.
The feature in the article seems to be for a slowly moving copter. I intend to use this on a plane so that the plane pitches up when obstacles are detected. Will ardupilot currently support this? I’m guessing not.
I noticed Randy was using a ZED 3D camera connected to an NVidia TX1 running OpenKai. That’s a good example of the nonsense and high expense that people normally have to deal with with stereo cameras. In my prototype, the FPGA is ~$100USD and the cameras are a few dollars each. I would say that it is a big difference vs. Randy’s setup. Not only that, but the power consumption must be immensely different between the two setups as well. I wish Randy could chime in here and help with some of the initial hand-holding to get this integrated.
Typical stereo camera systems:
Expensive
Usually slow (100ms latency or more)
Hi power usage
Bulky if onboard compute is needed
FPGA systems:
less than $200
Latency of 50ms or less. (The depth solution is ready as the image lines come out of the camera. It is not even one frame behind.)
Very low power, typical of FPGAs
small form factor (assuming the screen is not used / detachable)
Hi, this looks very interesting! Do you have a detailed article on how you built this? I would love to know!
The best use IMHO for this would be to replicate some of the work we have been doing the Intel RealSense cameras: https://youtu.be/-6PWB52J3ho
This at the moment will only work with Copter and Rover
What that would involve is:
Figure out 3-D points from the depth disparity (pretty easy to do once you know depth and focal length)
Figure out which 3-D points are important (you can filter them by how close/ how far away object is, or do object detection algorithm or anything you want)
Pack this 3D point into a new mavlink message OBSTACLE_DISTANCE_3D
Send it to ArduPilot, and set Parameters PRX_TYPE = 2, AVOID_ENABLE = 3, OA_TYPE = 1 , and you have a working obstacle avoiding copter/rover!
@rishabsingh3003@Hoehenarbeit I was thinking that it might be less work to trick ardupilot into thinking this stereo camera was one of the supported laser range finders. In this way, I may not have to modify any ardupilot code, and should be able to get the desired effect of the plane automatically ascending when terrain is getting in the way (off in the distance).
Hi @David82 ,
This is interesting project, but I am still not clear what do you mean by FPGA here? From the video you used dual camera (means Stereo camera). So what is the Fundamental difference between your “FPGA” system and Intel Real Sense system? You also mention about using range finder (lidar), will you use several lidars (or 360 deg. Lidar) to do “Visualisation” of the area (a kind of Lidar Mapping)?
The real sense camera looks like it also computes depth, but then ardupilot will have to process that depth image and figure out what action to take. That may be too computationally expensive/slow/laggy.
An FPGA is an different kind of “processor”. It is actually not a processor. …It is best to just google what an FPGA is.
I did not say that I will use a range finder. Have a look at my previous post, above.