Is It possible to integrate Nvidia's jupiter notebooks with Ardupilot for a Machine Learning Application?

ArduCopter Copter 4.0
1

Hi guys

I recently found all the Nvidia Jetson developer notebooks. Among them are collision avoidance, object detection etc. The one with collision avoidance is thought to be used for a jetbot. It occurs to me that this last jupyter notebook could be implemented on a drone with a camera and a pixhawk FC.

Is it possible to do what I’m proposing here ? and If so how should one start to integrate both sides( FC and CC to run the nvidia’s collision avoidance notebook ) ?

This video illustrates what I would like to do but on a drone instead: https://www.youtube.com/watch?v=U3VJCSDqdG4&t=361s

Thanks in advace.

1 Like
2

The short awnser is yes, but you will need the jetson on the drone (if that wasn’t already obvious to you).

You can send commands to the autopilot through USB (or directly from UART I suppose) from the Jetson or any other companion computer.

You can send mavlink commands with the python dronekit api.

3

Well first of all thanks for the response.

Yes I do have a Jetson nano and a S500 drone frame with a pixhawk mini. I am relatively new in this field so some of the questions that I might ask may seem too simple. Is there a protocol or a framework that allows you to run multiple scripts all at once. The reason I ask this is because from my understanding there are going to be about 3 different chunks of code running simultaneously.

  1. The jupyter notebook from Nvidia that is constanly checking whether there’s an obstacle or not
  2. The trajectory that the drone is following in which it may encounter some obstacles
  3. The move-aside code for when there is indeed an obstacle and the drone has to go around it until it can safely resume its path

I hope what I’m asking makes sense. or If you have a better alternative that I can look into …

Cheers.

4

When you say runny multiple scripts, are you asking about on the Jetson itself? You can run scripts directly on an autopilot but on EXTREMELY simple stuff written in LUA, nothing like this.

I doubt you will get very many responses on this forum about jetson / juipter notebook questions, I certainly cannot awnser anything specific about coding on them. I have been around other using the for this exact purpose but have not been involved with any of the software.

I know you said you are new, have you used Ardupilot/ Arducopter before?

5

Yes I mean the jetson because it is going to be the one controlling the drone’s trajectories.

Yeah no worries whatever kind of help in the form of information that I can get from here is much appreciated. This is for my university thesis so I have to know all its aspects.

I have used it in the nano for some Ros/gazebo simulations but haven’t flown a drone yet.

6

Hello,

I guess you are planning to use the underlying code of Jupyter, that is probably python ?
In that case @LuckyBird have written pretty good scripts for interfacing vision based system with ArduPilot. And @rishabsingh3003 made many improvements on the avoidance library using D435 & D455 Realsense cameras. Both can be reached on the vision channel.

The Nvidia JetBot is using a crude avoidance made for wheeled vehicle. The avoidance library can offer better strategies and we welcome new models implemenation. Depending on what camera ou plan to use , for example: a depth mapping camera like the Realsense could make a great system but you need to invest in building a specific data set as there are not many hanging around.

2 Likes
7

Oh ok Thanks for the advice. I will read through all that documentation and see what work best for me.

Cheers.