Greetings, I would like to listen to some opinions and criticism about this project that I am working on. It is an autonomous boat running Rover 4.0 on a Pix32 with a Raspberry Pi as a companion computer; it uses a camera taking images of the front of the boat and with a convolutional neural network detects and classifies obstacles like on the following images:
It uses the classification capabilities to create a different method of evasion for each type, like, if it detects a bird, it will sound a horn and won’t evade since they move out of the way themselves, but if it detects a buoy or a boat, it will measure the width of the obstacle and evade it, if the obstacle is a rock, since they are bigger underwater and the boat could run aground, it will evade with additional security distance from the rock. The mean average precision of the object detector is 0.93. The speed is 1.2 FPS.
It uses two consecutive images of the obstacle and the thin lens equation to measure distance from the camera to the obstacle, it is not capable to detect obstacles moving towards it from the sides. The distance measurement has a mean absolute error of 34.3 cm (13.5 inches) and for the measurement of the width of the obstacle, has a mean absolute error of 8.2 cm (3.2 inches).
The method of evasion is made so it can return to the original path as soon as possible, because the boat is intended to be used for bathymetry. The process is to make a simple square, it will detect an obstacle and stop 1 meter before reaching it, then turn 90° right or left until it moves 1 meter plus the width of the obstacle, then turn 90° towards the original destination, at that point it will turn the camera towards the obstacle and check its presence while advancing, when the obstacle is left behind enough to not appear on the camera, the boat moves 90° degrees towards the original navigation line until it reaches it and finally turns towards the original destination, like on the following image:
Seems like a great project to be working on. The simple square avoidance seem awfully inefficient though. How about heading to a point directly to the side of the obstacle and once at that waypoint head straight for the destination?
Or do you have a coverage requirement and must resume the original straight line course as much as possible?
I don’t have any useful input, but I am quite curious about how you run the CNN on the pi. Do you run the network on the CPU of the pi? How fast framerate do you get?
This is really great!! One of the difficulties with doing object avoidance on boats is that lidar or 3D camera often end up “seeing” the water which leads to unnecessary stops (simple avoidance) or turns (from “Bendy Ruler” path planner)… so I think what you’ve done is really useful.
By the way, it might be good to try sending in the distance to the obstacle to the flight code using either DISTANCE_SENSOR or OBSTACLE_DISTANCE mavlink mesages (the latter is probably better). Then you could leave the path planning around the obstacle to BendyRuler.
Thanks for your reply. Yes I forgot to mention that, the purpose of the boat is to use it for bathymetry, so it needs to travel on a straight line and, when evading, it must return to the line as soon as it can, so it doesn’t lose too much data of the depth
I’m using Tensorflow CPU to run the network on a Raspberry Pi 3 B+, I get a constant 1.2 FPS.
If I had the resources to get a Pi 4 and a Coral USB Accelerator, I could get around 33 FPS
Thanks for your feedback, I have been reading about the BendyRuler and it looks awesome, I didn’t know that it also worked with cameras, I thought it was for lidar. I also wanted to ask, does the algorithm tries to return to the original navigation line or it goes towards the waypoint after evading the obstacle? I forgot to mention that this boat is meant for bathymetry so it needs to go back to the transect.
Yes it will work with cameras or anything really if it can send in a mavlink message. @LuckyBird has been doing some work with the Intel RealSense 435i depth cameras so his scripts might be useful as a reference.
By default it will return back to the path but the OA_OPTIONS parameter can be used to configure it to head straight to the next waypoint. Perhaps the only risk is what happens when the obstacle can’t be seen anymore by the camera… the OA_DB_EXPIRE parameter controls how long obstacles are kept in the database though so if that were long enough (like 10 or 20 seconds) then that would give enough time for the boat to get around the obstacle even after it’s gone out of view.
What kind of CNN architecture and training set are you using? To get faster frame rates with same hardware you could use quantization (if you are not) and/or pruning your training set to only relevant images and reducing your network size, for instance your network does not need to recognize several kinds of cats, or elephants or cars (if you are using such a training set).
That said it sounds like 1.2 fps is good enough so perhaps it’s not worth it.
Thanks for your reply. I am using ssd mobilenet v2 trained with the COCO dataset, however I had to use transfer learning to train it to detect only the obstacles that I needed. Quantization looks like a very good idea that I could apply with tensorflow lite, thanks a lot.
If I may, what would really have a major impact on the operational use of autononous boats is the capability to DETECT THE COASTLINE.
Objects in the middle of the lake or river are of course a problem, but the real problem affecting “real world” surveys is that the actual coastline is always unknown. Satellite images such as Google Maps are not accurate enough, and for sure never updated. For example on hydro dam reservoirs the level of the water varies several meters during a single day, and as a consequence even the coastline changes…
If one could deploy an obstacle detection algorithm ( with cheap sensors such as vision ) , this would make a big difference. It would also enable navigations modes such as “TRACK THE SHORELINE AT X METERS DISTANCE”
I did some exchange with a University in mu Country, but the shoreline problem is still unsolved.
If it helps I can provide test cases, and are available also to do validations on the field
Thanks for your answer, that problem looks very interesting, I think that an edge detector on images from a camera could be an approach to detect the shoreline. Measuring the distance looks very hard to achieve but fascinating to give it a try; maybe measuring the speed at which the boat advances or distance traveled and comparing it with how much the line moves downwards on each image taken could be used to formulate an equation to measure the distance, I would need to investigate further.
I believe taht to make it robust you need to merge other information to support the visual estimates.
Speed is available, either measured or the planned one in AUTO mode, such as measured pitch and roll.
Of course the higher the point of view of the camera the better, I guess.
Maybe one could use some reference point on the aft part of the boat, making sure they are captured by the camera, so to be able to calibrate the whole thing and get absolute measures out of the algorithm
If one could at least get a trigger when the boat is closer than 10 meters to the shore (for example) it would already be very usefull.
Of course a continuous measurement even better !
Hi @rmackay9,
We have a problem with returning to the original path in copter 4.1.0 dev.
The problem is that by default(in our case), the vehicle goes in a new path to the waypoint after avoiding the Multi-rotor or being diverted by the wind, which is not desired, And this is the opposite of what you said before.
Note that we don’t have the OA_OPTIONS parameter at all !!
How can we fix the problem?
What is your solution?
Thanks and regards
Sorry, I missed this email somehow. I think if you’re not seeing the OA_ parameters at all then the issue is that the “minimized features” firmware is on the autopilot. If this is a Pixhawk then this can be solved by manually downloading the firmware from here: https://firmware.ardupilot.org/Rover/stable/Pixhawk1/
If it’s just the OA_OPTIONS parameter that is missing then it’s just that you’re using the stable version of the software and some features are only in “latest” (aka master). If you’re happy to use slightly untested software then you can load “latest” from the MP’s Firmware Install screen after pressing Ctrl-Q.
Hi
Thanks a lot
I need OA_OPTIONS for the Copter and my firmware version is also a 4.1.0 dev copter.
I also checked version 4.0.5 and I couldn’t find it there.
Thanks and regards.
Ah, OA_OPTIONS is only a valid parameter for Rovers and Boats. The ability to return to the original path hasn’t been implemented for Copter yet I’m afraid but I’ve added a to-do for it here: https://github.com/ArduPilot/ardupilot/issues/16095
