Tightly-Coupled Post-Processing of IMU/GNSS data

This is a project that I would like to initiate, and I am in search of a few talented developers to assist me with exporting INS/GNSS data for import to Novatel Inertial Explorer for tightly-coupled post-processing. In my mind, this serves two primary purposes; first, to use the trajectory to directly georeference images or LiDAR points on the earth for mapping, and second, to help improve or cross-compare ardupilot’s navigation algorithms with existing methods of post-processed state estimation. I believe the hardware is 95% there, but the process to get what we need isn’t. Ultimately, I’d like to see raw IMU and GNSS data extracted from the .BIN files in a format that works with Inertial Explorer. While we could use a standalone hardware device for this job, I think it would be wonderful to leverage the existing hardware on vehicles to do amazing stuff!

Novatel’s Inertial Explorer manual states that “In theory, virtually any IMU sensor can be used with Inertial Explorer. The only requirement is that the data be logged in the format provided in this section.” In addition, Inertial Explorer supports uBlox data. The format for these exports is here. https://docs.novatel.com/Waypoint/Content/PDFs/Waypoint_Software_User_Manual_OM-20000166.pdf

Inertial explorer should work just fine with the .ubx or RINEX files that can be generated from the logged ZED-F9P data, but we must make sure to get the correct messages in the GNSS log. I’m not sure exactly what those are, but I know ardupilot can configure these uBlox receivers accordingly. I plan to use a uBlox Zed-F9P through serial logging at 10Hz with RAW GNSS data recording turned on. My camera will send a pulse to the EXT_INT pin on the GNSS receiver that should generate a TM2 message. That message must certainly be recorded, and it currently isn’t. https://github.com/ArduPilot/ardupilot/issues/14400#issuecomment-694110125

Next, I’m concerned about clock drift on the autopilot boards. I’ve tested it and found it to be significant (tenths of seconds over a few hours, and my missions will be long). My understanding is that the way to correct clock drift is to use a PPS signal from the GNSS. I believe we may need some new architecture here to enable the proper timestamping of the 200Hz IMU data. That feature request is here: https://github.com/ArduPilot/ardupilot/issues/10369

This may be a paid project, and I am interested in having these features submitted as PR’s to ardupilot’s master branch so that this community can leverage this revolutionary technology for mapping or any other reason to extract the most accurate trajectory information available. If we do this right, we may be able to make accurate maps with very inexpensive hardware without having to conduct aerial triangulation, especially where pixel matching struggles. I think the relatively low-cost hardware on the market today can do some awesome things. I have Inertial Explorer and the hardware to test now. Please let me know how you could help and what it would take for me to work with you to get it done!

5 Likes

To have a proper solution you want to have a PPS time synchronized IMU.
For example the ADIS 16475 has a mode called external sync (Scaled Sync Mode is able to handle the PPS coming from the F9P): https://www.analog.com/media/en/technical-documentation/data-sheets/adis16475.pdf
We have experimented with it in the past.
Some background reading from KTH about theoretical results can be found here:
https://www.diva-portal.org/smash/get/diva2:1354856/FULLTEXT01.pdf

What we want to have and what we currently have access to are currently two separate things. I’m going to postulate that post-flight adjustment of the timestamps is possible, and we’ll try that first because we have the hardware and mechanisms to do that. Let’s assume we collect at 200Hz and the time drift is 0.1s per hour. We would be off by 20 samples in an hour. In otherwords, that’s one sample every 3 minutes. I don’t think it’s going to be an issue to squeeze or stretch the current sample timestamp a little bit to make it more accurate. We’re talking about 0.005s vs 0.0050277s, and I think as long as we provide inertial explorer with the correct timestamp, we should be good.

Now what you bring up is a good point with what may be necessary for live navigation and EKF consumption, but that’s separate from my needs here now.

The problem what is not easy to solve without physical (sync) connection between Pixhawk and F9P is the fact that IMU data is synced to processor time (say 200 Hz), while the F9P data inside the Rinex file is GPST ~+/-0.01 s, but when sending data to the Pixhawk on a serial bus it arrives after reception with ~0.2 sec delay. The F9P internally is able to determine GPST but it does not have clock steering function, hence sometimes it slips 0.01 sec, but more or less running parallel with GPST. The Pixhawk on the other hand has a cheap internal oscillator, what is sensitive to temperature, so after boot you could see a slow thermal drift & if you fly with your a/c you will see also change in temperature and an additional factor for diverging clocks. They can be aligned more or less sufficiently, but tight coupling tends to be rather sensitive for clock sync.

If you want to play a bit with some open-source tightly coupled GNSS/INS integration I recommend https://github.com/kaichen686/GINav

We already have physical accurate time sync through the Camera shutter feedback feature in ardupilot. I can either use just shutter feedback events as sync points, or I can use a PPS output from the GNSS as a consistent pulse to keep the sync regulated. 0.01s isn’t going to make or break this synchronization, and it all can happen in post-processing with adequate filters.

We did not have too much luck with connecting the flash hotshoe to both Pixhawk and F9P at the same time. We worked on it a bit but I guess we have not put too much effort in it. If you make it work and share logs (plus level arms, etc.) I would be happy to run some tightly integrated solutions (my PhD student has some in-house code, as well as running GiNAV).

Other solution what we have not explored is to route flash–>f9p (extint) and f9p PPS → Pixhawk. With this you could have a regularly spaced sync between GNSS and IMU, while the camera events are logged down to 20 ns accuracy.

I have the reach m+ connected with hotshoe alongwith the cube. If that log would help, i can share.

This is wonderful. Thank you. Here’s a link to two flights from yesterday: https://1drv.ms/u/s!AunBK4rE1ZDChYdrvG9aAXgV6t2OGA?e=5vaKef

Emlid Reach RS2 was the base station, Emlid M2 and a Sony RX1RII on the Elanus Duo aircraft. Each flight was about 35 minutes and took about 1250 images. You should find everything you need at the link. For some reason on each flight, the M2 didn’t pick up a few of the trigger releases, but the mRobotics Pixracer pro did. The hotshoe feedback wire was split to include a wire to the servos header. It was obvious what images were missing geotags based on my excel analysis. I’ll probably raise the issue on the Emlid forum.

I also commented on an issue that I may have uncovered with the timing somewhere. https://github.com/ArduPilot/ardupilot/pull/8409#issuecomment-1046166294. This is an irregular difference between Global Time US and the ZED-F9P’s event times at shutter release.

Also, we’re saying the same thing.

I am away from the office this week, but as soon as I get back I will take a look!

1 Like

How do things look? Based on the conversation in the last dev meeting, I think we’re going to have uBlox TM2 support built into ardupilot soon, so that’s exciting to get these events synchronized!

Monday we took a look at the data - to fit into our own tightly coupled framework we have to do some massaging of the raw data.

We are on it, but cannot promise anything at the moment.

Hi!
Can you share the data one more time. It seems like I have not downloaded all the raw GNSS files, just only the APM and .xls logs. THX

The link hasn’t changed from above; I see all of the data there still. Link repasted here: Microsoft OneDrive - Access files anywhere. Create docs with free Office Online.

I got this error (tried to log in with multiple different accounts):

Let’s try this new link. Sorry for the trouble!! Microsoft OneDrive - Access files anywhere. Create docs with free Office Online.