We’ve only been working with these cameras for a couple of months but we are already quite excited about their potential. They are relatively light weight, provide 3D depth information and can run AI pipelines right on the camera itself. We’ve also found that interaction with the developers goes very smoothly because they are knowledgeable, helpful and responsive.
Rishabh is the real expert on this topic but from my point of view, there are two parts to successful object avoidance. The “estimation” portion (i.e. where are the obstacles?) and the “control” portion (i.e. how do we avoid them?). Within ArduPilot we have put a lot of effort into the control side and it works fairly well but the estimation has always been a challenge. Most autopilot CPUs are optimised for real-time processing and can’t do a lot of data/image processing which has led us to use either relatively simple filtering and/or has forced us to rely on companion computer algorithms which are difficult to setup. I suspect these OAK-D AI cameras are going to be an important tool in helping improve both the setup and reliability of the “estimation” problem.
I wonder if luxonis has planned some kind of support for visual odometry? If I am not mistaken right now it is somehow straightforward to do the depth perception part with oak cameras, but it isn’t to do visual odometry correct?
I am interested in this, and I guess a big part of the potential users as well, to use this camera for non gps navigation, specially now that t265 is discontinued.
I would love if luxonis could provide more insights about this.
We have been working a lot of time with t265 in no gps setups, so if this is a possibility we would love to help to get the oak cameras to do this as well.
@rishabsingh3003 told me that it doesn’t yet officially support SLAM or visual odometry but someone was working on it. Certainly it could be done if all the depth data was sent to a companion computer but it would be nice if it could all be done on the camera as it is with the T265. My understanding (again from discussions with Rishabh) is that the the Movidius chip is really customised to run AI processing and so you can’t just run regular C++ on it… which makes me think it will be difficult to get the visual odometry working… but I hope I’m wrong.
I totally agree that what we really want is a single light camera that can do both the VIO and object avoidance… I also somehow hope the OAK-D can do this but in any case, that camera will eventually appear I’m sure.
@Luxonis-Brandon this looks great! I’ve just ordered two of the lite version on the kickstarter. It would be great if we could use this via the lua scripting in ArduPilot without the need for a companion computer
UART would be easiest, but CAN would also be a very nice option. We could add USB host support to ArduPilot and connect over USB (or oak could add USB host support?). USB connectors don’t tend to be great with vibration though, so UART would be preferred.
I hadn’t actually realised the lite doesn’t have flash to store it’s own fw when I ordered it - I got too excited by it and didn’t read up first
So I’ll probably use a PiZero or similar when they arrive, but it would be very nice to create a variant meant for drones, focussing on obstacle avoidance and VIO. We could feed good quality IMU data and height estimate from the flight controller to the oak to assist with the VIO.
discord is good. Best people to suggest features for the sensor side is @rishabsingh3003 and @rmackay9, with focus on obstacle avoidance and VIO.
Wolfsbane on your discord suggested we could try using the lite with fw and model on ArduPilot microSD. That could work if we added USB host support to ArduPilot and the transfer speeds we get are good enough. The downside is you couldn’t record the raw images for analysis when something doesn’t work right as we would not have nearly enough bandwidth on a STM32H7 for image transfer.
I’d need info on exactly what USB protocols and messages are needed to bring up the lite as a USB client.
Fantastic; thanks! So I just made an ArduPilot channel dedicated to this in our Discord. I think we can quickly iterate to something awesome here. And FWIW, we’ve had customers (closed source, unfortunately) take our SPI API and interface with the STM32H7 using our OAK-SoM-IoT. So it’s definitely doable.
Oh and since we have on-OAK encoding, that helps a TON. As we can downscale an image to say 640x400 and also encode it (JPEG, MJPEG, or even say h.265 if we want to get fancy) and then send it over a low-speed protocol like UART/SPI/etc. And when downscaled and encoded, things get really-tiny really fast.
Anyway, excited to make this a thing! I think will be really cool!
@rishabsingh3003 is probably the best to answer but I think the OAK-D-IoT-75 is perhaps the best choice at this moment. This is what I’ve recently bought anyway…
By the way, @david_sastre made a camera mount for the Intel T265 (which also suffers when exposed to vibration) and I’ve been meaning to ask him if he could create a modified design of the mount for the OAK-D-IoT-75 for me (and everyone else).
Thanks @rmackay9 . So I just realized my post was terribly ambiguous. So I was actually meaning to ask which drone anyone here would recommend we get. (We’re total drone newbies.)
Oh and speaking of OAK-D-IoT-75. We are working on an enclosure for it (this is Brandon from OpenCV/Luxonis. It’s in tooling for it now.
It’s the middle one below (OAK-D-Lite in front, OAK-D-IoT-75 middle, O.G. OAK-D in back):
This will make it easier to mount. Tooling should be done in December, so around January we should have this enclosed version. Not quite as tiny as OAK-D-Lite, but it can run standalone which is nice.
Ah, and perhaps I didn’t read your question thoroughly. Rishasbh is using a Hexsoon TD650 in the video I think.
I normally use the smaller EDU 450 which flies well but can be a bit of a challenge to assemble the first time you do it. I also have an ancient 3DR IRIS (hard to find these days) and one of the Holybro copters found on our ready-to-fly vehicles page.
Very happy to hear @Luxonis-Brandon is helping us here! I will definitively give these cameras a try.
@rmackay9 I would be happy to design another mount for this camera, but unfortunately I don’t have one here. If I could find precise measurements I could, or I could purchase one directly, I want to test it anyway :). Is there any way to purchase it directly from Europe?
Thanks @rmackay9 ! (I had just worded the question really poorly).
And thanks @david_sastre ! So for OAK-D-Lite, it’s in pre-order (link above, it won’t let me post it again). And we do ship to Europe (from that and on our store).
@Luxonis-Brandon I mentioned your offer to try to design a camera that would be perfect for drones to Professor Rob Mahony at ANU (Australian National University in Canberra). I work with his group on various state estimation projects with ArduPilot. They have a new EqF based VIO system that they are keen to try on your cameras. Rob wonders if a call with you, @rishabsingh3003, @rmackay9 and myself would be good to decide on what features would be needed? The ANU EqF VIO will be open source. We’re prototyping it with a RPi4 and a greyscale cheap USB global shutter camera at the moment, but an integration solution would be best. Even better if we can run the VIO algorithm directly on your camera CPU and also provide obstacle detection.
If this interests you then let me know what timezone you are in and what sort of times work and I can try to tee up a zoom call with Rob and the others (likely including Pieter van Goor, who is working on the EqF VIO maths as a post doc).
a paper on the VIO:
A talk Rob and Pieter gave on the maths at the ArduPilot dev conference:
Rob wonders if a call with you, @rishabsingh3003, @rmackay9 and myself would be good to decide on what features would be needed?
Yes! Would be quite interested in doing this!
Even better if we can run the VIO algorithm directly on your camera CPU and also provide obstacle detection.
Yes. I think this will work quite well. One member of our team enjoys low-level vectorizing and optimizing of such VIO algorithms for vector processors - but not coming up with the VIO algorithm itself - so it’s a good pairing (both in terms of hardware resources and interests/capabilities).
And the technical resources look great! I’m in Colorado so Mountain timezone. And free to shoot me an email at brandon at luxonis dot com for coordination (I can send my calendly there which might help across timezones).
