Autonomous Boat using Convolutional Neural Network for obstacle avoidance

Blog
3

Seems like a very cool project!

I don’t have any useful input, but I am quite curious about how you run the CNN on the pi. Do you run the network on the CPU of the pi? How fast framerate do you get?

4

This is really great!! One of the difficulties with doing object avoidance on boats is that lidar or 3D camera often end up “seeing” the water which leads to unnecessary stops (simple avoidance) or turns (from “Bendy Ruler” path planner)… so I think what you’ve done is really useful.

By the way, it might be good to try sending in the distance to the obstacle to the flight code using either DISTANCE_SENSOR or OBSTACLE_DISTANCE mavlink mesages (the latter is probably better). Then you could leave the path planning around the obstacle to BendyRuler.

4 Likes
5

Thanks for your reply. Yes I forgot to mention that, the purpose of the boat is to use it for bathymetry, so it needs to travel on a straight line and, when evading, it must return to the line as soon as it can, so it doesn’t lose too much data of the depth

6

I’m using Tensorflow CPU to run the network on a Raspberry Pi 3 B+, I get a constant 1.2 FPS.
If I had the resources to get a Pi 4 and a Coral USB Accelerator, I could get around 33 FPS

3 Likes
7

Thanks for your feedback, I have been reading about the BendyRuler and it looks awesome, I didn’t know that it also worked with cameras, I thought it was for lidar. I also wanted to ask, does the algorithm tries to return to the original navigation line or it goes towards the waypoint after evading the obstacle? I forgot to mention that this boat is meant for bathymetry so it needs to go back to the transect.

8

Yes it will work with cameras or anything really if it can send in a mavlink message. @LuckyBird has been doing some work with the Intel RealSense 435i depth cameras so his scripts might be useful as a reference.

By default it will return back to the path but the OA_OPTIONS parameter can be used to configure it to head straight to the next waypoint. Perhaps the only risk is what happens when the obstacle can’t be seen anymore by the camera… the OA_DB_EXPIRE parameter controls how long obstacles are kept in the database though so if that were long enough (like 10 or 20 seconds) then that would give enough time for the boat to get around the obstacle even after it’s gone out of view.

2 Likes
9

Don’t need a pi4, we get around 24 fps with a pi3 and a coral. We use it for target tracking. Nice work!!!

2 Likes
10

Cool! I guess 1.2 FPS is fast enough if the boat is moving slow enough.

1 Like
11

Great work!

What kind of CNN architecture and training set are you using? To get faster frame rates with same hardware you could use quantization (if you are not) and/or pruning your training set to only relevant images and reducing your network size, for instance your network does not need to recognize several kinds of cats, or elephants or cars (if you are using such a training set).

That said it sounds like 1.2 fps is good enough so perhaps it’s not worth it.

12

Thanks for your reply. I am using ssd mobilenet v2 trained with the COCO dataset, however I had to use transfer learning to train it to detect only the obstacles that I needed. Quantization looks like a very good idea that I could apply with tensorflow lite, thanks a lot.

1 Like
13

Hello,

Really an amazing work ! Well done

If I may, what would really have a major impact on the operational use of autononous boats is the capability to DETECT THE COASTLINE.

Objects in the middle of the lake or river are of course a problem, but the real problem affecting “real world” surveys is that the actual coastline is always unknown. Satellite images such as Google Maps are not accurate enough, and for sure never updated. For example on hydro dam reservoirs the level of the water varies several meters during a single day, and as a consequence even the coastline changes…

If one could deploy an obstacle detection algorithm ( with cheap sensors such as vision ) , this would make a big difference. It would also enable navigations modes such as “TRACK THE SHORELINE AT X METERS DISTANCE”

I did some exchange with a University in mu Country, but the shoreline problem is still unsolved.

If it helps I can provide test cases, and are available also to do validations on the field

Thank you
Regards

2 Likes
14

Thanks for your answer, that problem looks very interesting, I think that an edge detector on images from a camera could be an approach to detect the shoreline. Measuring the distance looks very hard to achieve but fascinating to give it a try; maybe measuring the speed at which the boat advances or distance traveled and comparing it with how much the line moves downwards on each image taken could be used to formulate an equation to measure the distance, I would need to investigate further.

15

I believe taht to make it robust you need to merge other information to support the visual estimates.
Speed is available, either measured or the planned one in AUTO mode, such as measured pitch and roll.
Of course the higher the point of view of the camera the better, I guess.
Maybe one could use some reference point on the aft part of the boat, making sure they are captured by the camera, so to be able to calibrate the whole thing and get absolute measures out of the algorithm

If one could at least get a trigger when the boat is closer than 10 meters to the shore (for example) it would already be very usefull.
Of course a continuous measurement even better !

1 Like
16

Hi @rmackay9,
We have a problem with returning to the original path in copter 4.1.0 dev.
The problem is that by default(in our case), the vehicle goes in a new path to the waypoint after avoiding the Multi-rotor or being diverted by the wind, which is not desired, And this is the opposite of what you said before.
Note that we don’t have the OA_OPTIONS parameter at all !!
How can we fix the problem?
What is your solution?
Thanks and regards

1 Like
17

I didn’t know that distance can be solved based on a single fixed camera, could you please share a bit more about your calculations?

18

@J_Esmaili,

Sorry, I missed this email somehow. I think if you’re not seeing the OA_ parameters at all then the issue is that the “minimized features” firmware is on the autopilot. If this is a Pixhawk then this can be solved by manually downloading the firmware from here: https://firmware.ardupilot.org/Rover/stable/Pixhawk1/

If it’s just the OA_OPTIONS parameter that is missing then it’s just that you’re using the stable version of the software and some features are only in “latest” (aka master). If you’re happy to use slightly untested software then you can load “latest” from the MP’s Firmware Install screen after pressing Ctrl-Q.

1 Like
19

Hi
Thanks a lot
I need OA_OPTIONS for the Copter and my firmware version is also a 4.1.0 dev copter.
I also checked version 4.0.5 and I couldn’t find it there.
Thanks and regards.

20

Hi @J_Esmaili,

Ah, OA_OPTIONS is only a valid parameter for Rovers and Boats. The ability to return to the original path hasn’t been implemented for Copter yet I’m afraid but I’ve added a to-do for it here: https://github.com/ArduPilot/ardupilot/issues/16095

1 Like
21

I have pi3 cameras with cables i bought never use i can donate to you for your cause Javo

22

This is a grea project. I am currently also working on a autonomous boat for bathymetry. I am looking for obstacle detection currently. Main goal is to detect buyos and the sailing boast fixed teher. Are more technical details of your project available in the web? Would love to take this approach as well. Which camera are you using? Or would you recommend usage of other technology after all your experience?