EKF Failsafe - f9p rtk corrections causes gps glitches

I’m testing a Holybro H-RTK F9P Rover Lite receiver.

My goal is RTK operations. I’m using a F9P receiver on my base - and sending corrections with Mission Planner. (stable version - not beta)

On my first test flight with RTK corrections, the copter achieved “FIXED” accuracy as soon as the copter gained about 10 meters altitude. I continued the climb to 100 meters and never reverted from “FIXED” TO 'FLOAT".

About half way up in the climb, a GPS glitch occurred but recovered quickly. Later in the climb more glitches occurred that eventually resulted in an EKF Failsafe.

I’ve conducted about 10 flights without RTK corrections, and receive no glitches.

Looking through the parameters, the only possibility I could think of was the baud rate. The Holybro docs say that the receiver defaults to 115200. The mavlink messages in the BIN log show that the F9P connects using 230400 baud. My serial3_baud was set to “38” for 38400 baud. I’m guessing that the ublox parameter for baud on the receiver is set to 230400 and ArduPilot simply adjusted to that.

To be sure there weren’t issues with a mismatch, I reset serial3_baud to “230” so it would match the receiver’s baud rate.

A subsequent test flight resulted in numerous gps glitches that would likely have resulted in a EKF failsafe had I not discontinued the flight.

The only other change I made was I changed the FS_EKF_ACTION from “LAND” to “ALT-HOLD”.

I continued making test flights without RTK corrections - and received no GPS glitches.

This Holybro F9P receiver is a new piece of equipment, and I had to move the wires on it’s 10-pin GH connector to a 8-pin GH connector (removing the 2 safety switch pins) so that it would connect to the Cube’s carrier board. So my first suspicion was the receiver, the connection, or the configuration.

But since there are absolutely no glitches in non-RTK operation, I’m suspecting a problem in the Copter firmware - or possibly, Mission Planner.

I’ve uploaded bin files to Dropbox. They all have the option set to record before arming.

First test flight with RTK corrections - resulting in EKF failsafe:

Second flight with RTK corrections showing multiple GPS glitches:

Sample test flight without RTK corrections - no GPS glitches:

Thank you for any comments or assistance!

ADDENDUM:

I conducted another test of flight with RTK corrections - but this time using Qgroundcontrol. No change in the situation - if anything, there were more GPS glitches.

I also noticed that with out RTK corrections the Holybro F9P receiver on my copter gets a consistent HDOP of 0.5. With the RTK corrections this jumps up to 0.8 or 0.9 - and occasionally jumps up to 1.5. The GPS glitches are associated with the high HDOP.

I suspect poor fixed base performance vice any firmware bug or issue. If you can access an NTRIP server to provide RTCM3 for performance comparison, that may prove useful.

A quick peek at one of my logs shows HDOP perfectly steady at 0.52 for hours with both GPS modules in an RTK Fixed state, likely indicating an extremely precise fix with a small but stable accuracy error.

Using the exact same base (ArduSimple F9P & Mission Planner) I had no GPS glitch issues with RTK corrections when I tested with a Here3 on the copter. That receiver struggled to go from “float” to “fixed” but with the same base as I’m using now, it never experienced GPS glitches.

So my feeling is that it’s doubtful the problem is with the base. And as I had the same problem with QGroundControl, probably not Mission Planner either.

There are two differences with the setup experiencing the GPS glitch problems: 1) now using a F9P on the copter, and 2) the F9P on the copter uses a serial connection instead of CAN.

There’s little to no information I’ve been able to find using F9P receivers on both ends with Mission Planner for RTK. So you may be right - it could be anything.

I tried using an Ntrip service when I first tested RTK with a Here3. Mission Planner would connect to the Ntrip Caster, but would never transmit connections to the copter. That was with the stable version of Mission Planner - I haven’t tried it yet with the beta.

What GNSS receivers are you using on your base and vehicle? If you’re using uBlox M8 receivers it may be an apples to oranges comparison with my setup.

I use all F9Ps and/or NTRIP + F9Ps, all with similar results whether using Mission Planner or direct radio injection of RTCM3.

That you couldn’t get NTRIP working suggests a possible comms issue.

You should be using the beta version of Mission Planner for these features. Its stable version is very outpaced by autopilot firmware development.

Yuri -

I appreciate your trying to be helpful. Did you look at the logs? Since you didn’t respond to my question about your use of an F9P receiver on your vehicle, can I assume that you haven’t?

While hunches are helpful - I’m afraid is has an effect you may not have considered. When others look at this thread and see a lot of responses from you, they may think that the problem is being addressed - and not take the time to contribute. So if you can’t offer something constructive, it may be better to stay silent.

The reason I hadn’t tried Ntrip is because there are numerous comments about the stable version of Mission Planner being buggy with Ntrip. But to clear up the concern you expressed, I loaded the beta and established an Ntrip connection. As the comments suggested - this worked and Mission Planner transmitted corrections properly using the Mission Planner beta.

Unfortunately, this didn’t change the result on my copter - as soon as I logged into the Ntrip Caster, the copter began experiencing GPS glitches.

This is the log file from that test.

What would be helpful would be to hear from anyone who’s successfully used an F9P receiver on their vehicle using RTK. Since the Here3-Pro is unavailable, I suspect few people have - so I expect any problems with that configuration haven’t yet come to surface.

I’m quite confident that my ground station is working properly. If you have time to look into the logs and see if anything is problematic, I’d appreciate that. While hunches have value, I’m hoping that someone knowledgeable about the firmware will have a few moments to look into this problem.

If it is a problem with firmware - the incidence of it will soon increase as more people install F9P receivers on their vehicles.

I looked at your logs. The GPS rapidly cycles between No Fix and an RTK fix state. There are no smoking guns in the parameters. It appears you are using a recent version of uBlox firmware.

Again, this suggests a problem with the RTK injection mechanism/comms.

I gave you the solution for NTRIP, which rules out a variable.

I answered your F9P question very succinctly.

I use all F9Ps and/or NTRIP + F9Ps, all with similar results whether using Mission Planner or direct radio injection of RTCM3.

To expand on that, I’ve used F9Ps on at least 9 vehicles, Copters and Rovers alike, and have troubleshot others’ problems with them countless times. I’ve helped discover and fix several bugs/anomalies in the uBlox driver and feature tested new driver options.

I also saw your post HOURS before responding. No one else did, so I offered what I could.

But apparently that’s just “trying” and “not constructive.”

My bad for missing your comment regarding your use of F9P’s.

If you could shed some light on your precise equipment and configuration, perhaps I can learn from your setup.

No doubt you believe you have helped many others. Perhaps you’ll be able to add this trouble report to your list of successes.

In the mean time I’m still hoping someone will offer some diagnostic assistance.

I’ve not noticed any changes to GPS/RTK handling in any of the changelogs to firmware since I’ve been involved. So my “hunch” is that as installing F9P’s on vehicles and ground stations is fairly new, some element of employing them may have something requiring a change.

I was sure hoping I had simply set a parameter incorrectly.

Exact configurations with which I’ve had success:

M8P fixed base direct USB connection to Mission Planner + MAVLink injection to M8P.
M8P fixed base direct USB connection to Mission Planner + MAVLink injection to F9P.
F9P fixed base direct USB connection to Mission Planner + MAVLink injection to F9P.
F9P fixed base UDP serial connection to Mission Planner + MAVLink injection F9P.
NTRIP internet connection to Mission Planner + MAVLInk injection to F9P.
F9P fixed base serial connection to SIK radio, SIK radio direct connection to F9P.
F9P fixed base serial connection to LoRa radio, LoRa radio direct connection to F9P.
F9P fixed base serial connection to ESP32, ESP32 direct connection to F9P, wifi protocol.
F9P fixed base serial connection to ESP32, ESP32 direct connection to F9P, ESP-Now protocol.

ArduPilot firmware versions have spanned Copter/Rover 4.0 through 4.3-dev.

The use of F9Ps isn’t as new as you might think, and the driver is fairly mature at this point, having been labeled “stable” through at least 3 major revisions. There were significant changes to it between 4.0 and 4.1. It has not seen much change in 4.2 and 4.3 because all testing indicates good/stable performance.

“No doubt you believe you have helped many others.”

No. That is a fact, not a belief.

Thank you.

All of your equipment citations are about fixed base. What are you using on your vehicles? Not sure if I mentioned but when using a Here3 on the vehicle, I did not have this problem.

Not sure if it matters - but it would be nice to know what brand and model receivers you’re using - not just the uBlox receiver type.

I know F9P’s have been around a while - but I’ve seen practically no mention of them in on-line conversations - and none about use in RTK operation.

With luck you’ll soon be able to add this to your list of successes.

What do you suggest I do to further the diagnosis at this point?

At the end of each line where I say “to F9P” my intention was to indicate that it is an F9P (or two) on the vehicle itself.

My M8Ps were HolyBro branded. All of my F9Ps are from ArduSimple (SimpleRTK2B and SimpleRTK2B+Heading).

Antennas have been either the magnetic uBlox branded patch antennas or (more frequently) survey grade antennas from ArduSimple (even on the Rovers themselves). I have not yet used helical antennas, though I have observed others’ success with them.

I suspect antennas are not your problem. The satellite count is quite high (upper 20s) and stable.

If you can take a laptop outside and connect it directly to a fixed base with a USB cable and to the autopilot with a USB cable, then run your RTCM3 via Mission Planner with those wired connections, we can eliminate the telemetry link as a possible cause. You won’t be able to fly that way, of course, but leaving it connected for 20-30 minutes in that configuration with LOG_DISARMED=1 may prove enlightening.

I’m also using the ArduSimple SimpleRTK2B as well on my base. I found a nice case for it on Thingiverse - works well as base receiver. I’m also using the SparkFun surveying antenna. (TOP106)

The SimpleRTK2B is connected via USB to my laptop I’m using as a base.

It appears the difference in our setups is that I’m using a Holybro receiver on my vehicle.

All of my logs have LOG_DISARMED=1. I think I mentioned that in my original post - my bad if I neglected to do so.

Please let me know if I can run additional tests and upload additional logs.

Can you confirm the F9P firmware version is 1.13 or newer?

1 Like

I installed the latest available firmware on the uBlox website on both my ArduSimple and Holybro F9P receivers.

In both cases I did not save and restore the parameters. (I saved them on the Holybro - but didn’t restore them)

So as I understand it, both receivers are using default parameters. And since the problem exists using Ntrip, only the Holybro (copter) receiver’s parameters might come into play.

As I understand uBlox parameters from using U-Center, changing a parameter puts them in volatile memory until a “save” is executed and the values are placed in non-volatile memory. I’ve found no mention of setting uBlox parameters for rover or base receivers to use them with ArduPilot or Mission Planner, I’ve always assumed that the ArduPilot firmware and Mission Planner will set any receiver parameters necessary at power up - and that they’ll not persist after power down.

It’s interesting about the Here3. There’s an option in Mission Planner to update the Here3 firmware - but I don’t believe it’s for the M8P receiver contained in the Here3 - rather for the Here3’s other processors. The update happens way too fast to be a uBlox receiver update. I suspect the only way to update the M8P firmware on the Here3 is by using CAN passthrough and U-Center. I tried to do that, but failed to get U-Center to connect to the receiver.

To answer your first post: since there are no issues when RTCM3 is not present, wiring is not a likely root cause.

The firmware and F9P configuration to date sounds very reasonable. Your understanding of u-Center and the GPS configuration-saving behavior is correct.

To be certain that the Copter’s GPS module is in its default state, it is advisable to connect to u-Center and use the screen shown below to “Revert to default configuration” (click Send) and then “Save current configuration” (click Send again).

image

I dug into your parameters again and compared them against a known working Copter configuration, which again turned up no clues.

Can you repeat the NTRIP test with a fresh boot of the autopilot, the fixed base physically disconnected from the GCS laptop, and a fresh instance of Mission Planner beta? (When I say “fresh instance,” I mean start MP and DO NOT connect it to any fixed base prior to activating NTRIP). I want to be sure we completely remove the fixed base from the equation.

Per your request:

Mission Planner beta applied this morning.

Fresh boot of the laptop - no receiver attached.

On this log, the copter was started - and the Ntrip connection was established. I waited a couple of minutes before connecting mavlink to the copter.

The GPS glitches show up almost immediately. I went to LOITER mode and armed the copter. I spooled up the motors, but didn’t lift off. I disarmed. I was receiving regular GPS glitches the entire time.

Then I shut down the Ntrip server and shut down Mission Planner and the laptop. But I left the copter powered up.

It took about 30 to 60 seconds before the GPS glitches stopped. Oddly, the fast blinking green line on the copter receiver remained - as it does when it starts receiving RTK corrections.

I then put the copter in the air for a minute - no glitches.

After landing the copter’s receiver was still doing the fast blinking green.

You have both SERIAL3 and SERIAL4 defined as GPS ports (protocol 5) but only one GPS connected.

Set the unused port to protocol -1 (none).

I’m not optimistic that this is the fix, but I want to eliminate all possible causes.

Also, did you try the direct/wired connection test that I mentioned earlier?

Just a fast chime-in - I haven’t looked at the logs, but played a bit with F9Ps. Try using only GPS and Glonass, on both base and rover. And set the survey-in period for the base - you have to configure the base as base (check Drotek’s site for some info) - for twice the time it’s set at now.
May be a problem of 5Hz position while correcting for all 4 constellations or a “jumpy” base.

Disabling the unused serial port (de-selecting “GPS”) did not change the problem.

However, the direct wired connection you suggestion does seem to have solved the problem.

I’ve always had the baud rate of my SiK radio RFD900’s set to 57,600 - per the ArduPilot docs.

Now I’m wondering if that’s simply too slow to handle the the F9P corrections data stream.

I’m going to try increasing the SiK radio baud rate up to 230400 - which is the baud rate of the rover F9P according to the mavlink messages. That may be too fast to maintain a stable SiK radio connection on the RFD900’s - but it should tell us if we’re barking up the right tree.

Do you happen to know how fast a baud rate the RFD900’s should be able to handle?

If this turns out to be true, then it opens another whole can of worms. In fact - it appears to be open already. If upon entering RTK mode there’s a breach in connection with the corrections data stream, the GPS simply loses its lock, instead of reverting back to non-RTK mode - the vehicle will enter a EKF failsafe if the telemetry is lost because the GPS will lose it’s lock. This could be very dangerous should there be a break in telemetry - and might support the justification for a secondary GPS.

You don’t need that much more headroom for corrections. I’ve ran NTRIP through 19200 baud LORA and I bet it could squeeze through 9600.