Precision landing with multiple AprilTag

chobitsfan · August 22, 2022, 2:48am

Inspired by @anbello 's work, @ppoirier 's work and @LuckyBird 's work.
GoodRobots created an excellent project for vision based precision landing. It use multiple fiducial markers. However, when landing on a limit space (for example, landing on a charging station or on a vehicle). Its markers arrangement makes landing spot off-center (drones will land on the smallest marker).
a1-landing

I have a different approach. By utilizing the ability of pose estimation of markers and LANDING_TARGET MAVLink 2.0 extension fields. We can have multiple markers in different size, but allow drones to land on center of landing pad. In this approach, each marker does not treated as the landing target. Each marker provides a landing target offset.

For example, if drones see marker id 1, then landing target is 0.2m down from marker id 1. Multiple small markers can be placed alone the edge of the landing pad, to avoid drones get lost during the final approach stage

My branch to test this function GitHub - chobitsfan/ardupilot at apriltag_plnd. A PR is proposed to support LANDING_TARGET extension field.

chobitsfan · August 28, 2022, 10:31am

RTL test, f450 frame, landing pad size 50x60 cm, largest marker is printed in a3 paper

kripper · March 16, 2023, 8:24am

Hi @chobitsfan,

I’m the current maintainer of RosettaDrone.

I came to your same conclusion about the need of an offset for each marker relative to the landing position, which also adds the ability to handle multiple “active” markers instead of only one at a time.

I’m also thinking in adding extrapolation and a PID controller.

I just posted some issues on the goodrobots vision landing issue section and I’m looking for a vision landing opensource project to contribute to.

I changed your branch, but I didn’t see the computer vision code for tracking the markers. Is it on another repo?
It’s there any documentation for setting up the environment?

chobitsfan · March 16, 2023, 3:10pm

Hi @kripper Thank you for commenting

I changed your branch, but I didn’t see the computer vision code for tracking the markers. Is it on another repo?

My apriltag plnd code GitHub - chobitsfan/apriltag_plnd , it use v4l2 for raspberry pi camera. I also try to use new libcamera GitHub - chobitsfan/libcamera-apps

kripper · March 16, 2023, 3:36pm

Perfect. I’m jumping in.

The link in your post (“My branch to test this function…”) should maybe link to this repo instead of your old already merged branch.

I will review and write in your issues section.

Maybe we can also convince @fnoop (from vision landing and Maverick) to merge both projects, since he wanted to switch to Apriltag.

kripper · March 16, 2023, 6:13pm

Discussion about merging projects continues here:

github.com/chobitsfan/apriltag_plnd

Merging with vision landing project

opened 05:03PM - 16 Mar 23 UTC

kripper

Hi @chobitsfan, I'm the current maintainer of RosettaDrone and we are looking… forward to contribute to an opensource visual based precision landing project. I reviewed your implementation and understood everything (code is clean). Your implementation has this pros: * Uses Apriltag * Computes a landing point (offset) independent of the marker centers. * Sends out the data to compute the yaw Vision landing's track_targets has this pros: * gstreamer support...but probably slower then just working with the latest decoded frame * Uses an input video queue to reduce latency (receives new frames in background while doing the image processing at the same time in other thread)...but I believe the V4L implementation does something similar under the hood (?). It's important to always have the next frame decoded in memory ready to be processed as soon as the previous detection process finishes. Unnecessary frames will be dropped (wasted resources), but the overall latency is optimized. * outputs AR images/video to debug. * threshold filter based on historical detections (tells if we can trust the detection). Does the Apriltag library output wrong detections? If so, maybe we can just trust all detections (?) and we don't need to filter again. Anyway, the filters should be applied using the results of multiple markers. * filter wrong pose estimations, for example, when a marker is detected with negative z value, ie. supposedly being behind the camera. * documentation (how to calibrate the camera for correct pose estimation) I believe both projects should be merged somehow so a maintainer community can be built around. Of course, they are completely different implementations, but the final goal is exactly the same. Before proposing a project merge strategy, I would like to see the code of the "frontend" where you process the marker data sent via IPC from the backend "capture.c". You are probably doing there stuff like filtering out detection errors, maybe generating AR images for debugging and other stuff that is done in the equivalent vision landing python script. About the scope: We are also interested in doing some extrapolation to: * Compute the absolute target position to deal with latencies: we need to track the drone position when the image was captured and add the displacement of the drone since this timestamp. * Correct the absolute target position considering the last computed absolute positions: since the target could be moving, we could do a linear extrapolation based on the last two positions or a b-spline extrapolation considering tree or more points. * Also a PID could be used to smooth the drone landing motion Anyway, I believe the merged project should just focus on returning the position of the target relative to the camera, **including the error filters required to compute a robust and consistent landing target position** based on all markers and maybe also considering previous computations. The rest I mentioned above could be implemented in the flight controller. BTW, do you know if this has been implemented? For RosettaDrone, we will need a independent implementation (apart from the FC), so it would be ideal to use a shared library for this (a second layer separated from the flight controller and from the target tracking/capture layer).

kripper · March 20, 2023, 2:15am

And here is the final result of merging @fnoop’s (GoodRobots) Vision Landing with @chobitsfan’s alternative AprilTag approach, plus other improvements:

Long live OpenSource