Easy way to integrate AI with ArduPilot? OAK-D! (Part 1)

Since I’ve ordered OAK Lite, I’ve started to getting familiar with OpenVINO and I’m disappointed with the ONNX support, because the docs state that supported opset is ver 8 (which is far behind the current one - 15!).

We’re more than happy to help out in our Discord as well on stuff like this. We’ve done this quite a bit for converting neural models/etc. EDIT: some examples here: depthai-ml-training/colab-notebooks at master · luxonis/depthai-ml-training · GitHub

1 Like

what is the power consumption of the camera? it looks like it has a lot of cooling.

It’s about 5W total power when running everything that it can run in parallel. So yes about at the limit of passive cooling but it does find.

@Luxonis-Brandon

I saw a post a few months ago where you and @rmackay9 were discussing the possibility of enabling this camera with the ability to do both VIO and obstacle avoidance. Is that still in the works?

Yes, we’re likely going to kick this off in the new year, starting mid January. We have wide FOV models coming out as well like OAK-D W as well:

And these will be more capable for VIO because of the wider FOV.

Depth-AI has already a feature tracker engine, you need the external software or a internal firmware I think. Many of that external software use Ceres,G2o,GTAM…and the one you named enter in that category I think, I don’t think is very probable that can run in a OAK camera.
About the internal option should be something like the feature tracker/slam of the T265.

About the internal firmware option I read something about that possibility soon , but Brandon is here so I guess he will give any news related with that .

Yes. Spectacular AI has done some impressive host-side VIO with DepthAI outputs as well.

Youtube MSBGzktEVQg

And yes we will be doing a direct-on-DepthAI VIO as well. Likely will be lower accuracy/etc. But a zero-to-one sort of thing. That will just allow VIO to be spit out directly from the camera for folks who don’t need absolute state-of-the-art but more around the performance of T265 (which is WAY below state of the art - but still useful).

And Spectacular AI is planning a better-than-state-of-art version that will run onboard our Series 3 OAK (which is coming late 2022) which is based on MA2095 (Gen3 Movidius).

Thanks and sorry about the delay.

-Brandon

2 Likes

Hrm… not sure if my posts went through. They keep getting flagged as SPAM. I was accused of being an AI at my last job (on many occasions) by people. Now AI is thinking I’m an AI. Now I think I must be an AI!

1 Like
  1. Will the code for your VIO be open-sourced? IMO closed-source was a major flaw of T265. Without access to code there’s no way to proper tune and fix bugs for your particular use case
  2. Can you say something more about Gen3 Movidius? How it’s compared with Gen2? I’ve found some press announcements from 2019 but now it’s 2022 and I can’t find anything on Intel website or anywhere else.
1 Like
  1. Yes, likely. Unfortunately the build tools can’t be open-sourced but ideally we can write it in a way that it can be cross-compiled to other architectures with open-source build tools.
  2. Yes. It’s about 10x the AI performance of Myriad X. So more like a Xavier NX in terms of AI performance. And it has a bunch of other niceties that make the CV/depth better. And in addition to that it has about the equivalent of a Raspberry Pi Zero 2 built-in (quad-core ARM 1 GHz) that runs YOCTO Linux. So on this version it will be easier to run a variety of host application on-chip.

Will are making our OAK-SoM-Pro with Keem Bay (Gen3 Movidius) as well.

Thanks,
Brandon

3 Likes

Hi @rishabsingh3003, could you write a bit about your ideas to improve poor depth quality?

Hello, i have a .tflite model we use on a coral for our photovoltaic job to keep dron in track. How should i proceed to convert it to use with oak-d lite? I tried to follow your links but i soon got lost.

thanks,

Corrado

Hi @Corrado_Steri ,

We can help on our Discord. I would post the link here but my posts are always blocked if I post a link.

So I’ll try to separate it out so you can build-your-own-link:

https colon slash slash discord dot gg slash luxonis

Erik can likely help you through it and/or Matija on there (look for Luxonis-Erik and Luxonis-Matija).

Thanks,
Brandon

And in case the link actually works, here it is:

Sorry but i am a bit discord dumb, i already have an account there and used it to join discord and accepted your invite than i tried to write and asked me for a phone validation wich i did but it told me the number is already used for another server, wich obviously is.

1 Like

Nevermind i managed it, my phone was connected to my old discord account, had to login with old credentials and eliminate old account, than all was good.

1 Like

One last thing it is not really clear to me. As i understand from this blog post i should get the fixed focus version but it looks like there is not fixed focus on the oak-d lite. Should i just wait or have any other advice?

Hi @Corrado_Steri ,

Thanks! Yes, I too have Discord troubles. You’re not alone.

So on OAK-D-Lite actually we do have a fixed-focus version. I’m not sure if the link will be stopped by the ArduPilot forum (usually it is blocked) - so if it is, please shoot support at luxonis.com an email and we’ll send the link to you.

But TLDR, juts go to OAK-D-Lite on our webstore, and then there’s a selector for RGB CAMERA FOCUS and you can choose Auto-focus or Fixed-focus there.

Thanks,
Brandon

And here’s an attempt at the direct-link:

Made a mistake i meant the IOT version isn’t the best one since has esp onboard and flash but i can’t find it fixed focus.

Oh yes we don’t have any fixed-focus OAK-D-IoT actually. As we discovered folks wanted fixed focus after that design.

We can build to order from the next batch though if you’re willing to wait for the next production round… likely in 3 months.

Oh and feel free to shoot me an email at brandon -at- luxonis -dot- com to discuss.

Thanks,
Brandon

Thanks, will get the fixed lite.

Corrado

1 Like