GSoC 2024: High Altitude Non-GPS Navigation

Blog
1

HIGH Altitude Non-GPS Navigation(1)
HIGH Altitude Non-GPS Navigation(1)1920×1080 249 KB

Introduction

Hello AP community! I am Sanket Sharma, a fourth-year undergraduate student currently pursuing B.E. Mechatronics from India. I am happy to announce that I have been selected to work with Ardupilot again under GSoC’24 with my mentors @rmackay9 and @tridge. In this blog, I’ll be discussing my GSoC project focused on High Altitude Non-GPS Navigation.

This project aims to develop a high-altitude non-GPS position estimation algorithm for UAVs, enabling them to localize and navigate in GPS-denied areas at altitudes of at least 100m using a downward-facing camera. The project involves extensive work with computer vision, image processing, and advanced control systems to achieve precise location estimation without GPS.

Current Scenario

Ardupilot is an open-source and versatile autopilot system that provides support for several vehicle types (rovers, copters, planes, and more), sensors, and frame types. It has multiple autonomous features like RTL, Loiter, Auto, Guided mode, etc., all of which depend on GPS as the primary source of position estimate. While there are non-GPS navigation techniques available, they are mostly limited to low altitudes and indoor environments.

High-altitude pose estimation in GPS-denied environments can be a game-changer in the UAV industry, allowing drones to remain stable and reliable even after GPS loss at high altitudes. This feature will not only improve user experience but also enhance safety for people and the drones themselves.

Terms

Definitions of the important terms:

  • SIFT (Scale-Invariant Feature Transform): An algorithm for detecting and describing local features in images.
  • FLANN (Fast Library for Approximate Nearest Neighbors): Used for matching SIFT descriptors between different images.
  • RANSAC (Random Sample Consensus): A method for robustly estimating a homography matrix by considering inlier matches and ignoring outliers.
  • Homography: A transformation matrix used to align features in different images, accounting for rotation, translation, and changes in perspective.
  • DSMAC: Digital Scene-Matching Area Correlator is a navigation technology that enhances accuracy by comparing real-time images with pre-stored terrain data. It enables precise location determination by matching current visuals to known reference scenes.

Approach

Though I have two approaches for the task:

  1. To have a map prior to the start of flight as a reference in case on GPS-Denied flights and use SIFT-FLAN (or AKAZE-FLANN) based image feature matching algorithm.
  2. Instead of having a map beforehand, we can calculate position based on an image matching and IMU back propagation based SLAM algorithm to estimate position of the UAV.

For GSOC, I have planned to go with the first approach to lay the foundation for developing the second approach.

I have divided the approach into 2 step tasks:

Step 1: Setting up SITL Simulation Environment and Adding the High Altitude Pose Estimation Feature

This task revolves around setting up the environment, dependencies, and configuring Ardupilot’s build system along with setting up a copter (software testing (SITL)).

  1. Setting up Ubuntu 22.04 for AP SITL
  2. Setting up Ardupilot SITL and Gazebo:
  • Recreating the environment for high-altitude position estimation in Gazebo. I have added a sattelite TIFF scaled image to replicate high altitude environments.

Screenshot from 2024-07-05 00-23-51
Screenshot from 2024-07-05 00-23-511915×1017 170 KB

  • Setting up the required parameters as mentioned here.
  1. Modify Ardupilot’s codebase:
  • A new parameter is planned to be added for this new feature/mode (e.g., NGPS_DATACOL for mapping purpose).
  1. Create an image stitching algorithm to create the map of the area. A SIFT based image stitching algorithm is made to stitch and form reference maps from the input images. Various image processing techniques like histogram equalization, etc, are used for seamless map stitching.

Screenshot from 2024-04-01 02-32-55
Screenshot from 2024-04-01 02-32-551658×524 91.3 KB

Screenshot from 2024-07-05 00-31-12
Screenshot from 2024-07-05 00-31-12845×583 71.7 KB

  1. Create the camera-based position estimation script using PyMAVLink and OpenCV. I have developed a SIFT-FLANN based position estimator using linear interpolation and extrapolation of known locations to map pixel coordinates to the actual locations. An AKAZE-FLANN based algorithm is also on checklist to compare with.

yt

Another approach that we can use is basic trigonometry using camera’s intrinsic parameters.

lagrida_latex_editor_x
lagrida_latex_editor(1)
lagrida_latex_editor

  1. Test image stitching and localization in Gazebo simulation: Test done in Gazebo as mentioned above

Step 2: Setting up the Vehicle and Companion Computer Environment

Here we focus on setting up the vehicle and companion computer for making high-altitude non-GPS positioning a success.

  1. Setting up FCU for copter:
  • Build the modified code and upload it to the FCU.
  1. Setting up the companion computer:
  • Modify the PyMAVLink code for the companion computer and enable image capturing through the gimbal.
  • Install all required packages on the companion computer.
  1. Parameters and Tuning the EKF:
  • Set base parameters and fine-tune EKF and the position estimation script for visual odometry.

Some of my work and scripts associated are mentioned here.

Optional Task: Developing GPS/Non-GPS Transition Lua Script

If time allows, a LUA script will be developed to handle the transition between GPS and high-altitude non-GPS positioning.

Conclusion

This project aims to significantly improve the stability and reliability of UAVs in GPS-denied high-altitude environments, providing valuable capabilities to the Ardupilot community. I am looking forward to contributing to the Ardupilot codebase again under GSoC and working with the community.

I would love to hear your feedback and suggestions in the comments below. Thank you!

Associated PR’s and repositories:

GitHub - snktshrma/ardupilot_gazebo_ap at arena-high-alt

5 Likes
2

把飞机想象成一个鼠标，然后它通过光学传感器记录位置变化量。

1 Like
3

你好 @1111153 ！谢谢你的建议。我还将尝试流算法，看看哪种方式可以获得更好的结果。

Hi @1111153 ! Thanks for your advice. I will also try the flow algorithm to see which way gives better results.

4

Theoretically, using a downward-facing camera to lock onto the x-y position seems feasible and relatively inexpensive. However, in certain special scenarios, it might be a better choice to appropriately integrate external sensors. In other words, I believe that using a single camera for such positioning might only be suitable for sunny conditions. When at high altitudes (relatively speaking), there will often be brief or prolonged cloud cover, making it logically unavoidable to fail in positioning via relative position. Over time, this will lead to significant x-y-z drift. In such cases, descending in altitude may no longer lock onto the original position. Therefore, I believe this issue requires a multi-sensor fusion approach (e.g., barometer, IMU, camera, etc.) to adjust the x-y-z positioning. Alternatively, can we first logically confirm which abnormal scenarios might occur and which sensors should be used in those scenarios?

2 Likes
5

Hi @lida2003 ! Thanks for the suggestions. Cloud cover and environmental distortions can really cause issues. As you suggested, a multi-sensor fusion approach might be necessary. I’m considering that implementing a UKF can handle these non-linear or challenging conditions, and also about other sensors that could be reliable in these situations. It’s definitely something we need to explore. Thanks again!

6

Logically speaking, when in high altitude, there are mostly two main issues:

  1. cloud cover (maybe long time)
  2. cloud movement （not constant）

So I think the code might use baro/windspeed/IMU(EKF/UKF) data for high altitude position hold.

A) baro/mag/windspeed/IMU(EKF/UKF) data: covered with cloud.
B) downward camera(health analysis) + baro/mag/windspeed/IMU(EKF/UKF) data: clear ground image.

And, those above data is Non-GPS. I’m NOT sure about the accuracy. Do you have any kind of these test data? Or is it possible to simulate moving cloud in gazebo?

BTW, high-altitude camera vapor might be an issue for image analysis.

EDIT: I think the accuracy is NOT good, but it should be something like Dead Reckoning Failsafe — Copter documentation, which is NOT that accurate. If tolerance range of the drift (along with time, simulation test) is accepted by the application.

1 Like
7

Hi @lida2003! Thanks for the detailed insights!

Yes, all those sensors will be in use in the current scenario as EKF will require them alongside vision estimates when there’s a clear ground image. With a clear ground image, I am getting pretty good results in gazebo right now (will post a video soon).

Regarding test data, I don’t have specific non-GPS accuracy data at the moment, but I agree that the accuracy might be limited and similar to a Dead Reckoning Failsafe.

For the cloud cover case, I guess it’s more about dead reckoning, as you mentioned, and it needs a deep analysis to solve this as IMU+windspeed estimation can only handle up to a very little time before drift starts to make things worse.
If that’s the case, we might have to implement a UKF on a companion computer as a state estimator instead of internal EKF to get state estimations for these type of conditions as EKF might not be able to cope up.(@rmackay9 sir might have a say on this UKF thing)

I’ll try to simulate clouds in Gazebo after the current implementation is done and will update here on the blog.

Vapor at high altitudes is another good point to consider for image analysis. I’ll proceed with these considerations and see how well the system can handle the drift within an acceptable range.
Thanks again for your insights!

8

Well, I’m NOT sure about how to define/test the accuray,

  • The drift distance away from the original position?
  • GPS coordinates v.s. current estimate postion? Is it possible for gazebo to record those values seperately?

I am interested in those test results and test variables (such as wind speed, cloud cover time, duration of cover, intermittent cover, etc.), as well as the subsequent actions taken when accuracy exceeds the threshold (especially how actions are executed when the accuracy requirements are not met, such as preparing for descend or landing, etc.).

EDIT: I think accuracy is related with altitude/image width,height/indicator x,y/angle of the camera when taking picture(Especially with wind, it may be necessary to move forward into the wind direction a bit more, ensuring the marker stays within the camera’s field of view.)

2 Likes
9

Hi @lida2003 !
Got it. Sure, I’ll be testing the accuracy of drift and gps vs estimate position in gazebo as well as video from a drone from an actual flight if I can.

Got your point. These action plans and failsafes are much needed at high altitudes. I will keep you posted with the tests and will tag you when I post some videos and test results herein.

Thanks for your insights. Would love to know much of your insights for the project. I highly appreciate that @lida2003 !

1 Like
10

Hi there,
Here’s a video of initial tests done on the simulator.