I appreciate your trying to be helpful. Did you look at the logs? Since you didn’t respond to my question about your use of an F9P receiver on your vehicle, can I assume that you haven’t?
While hunches are helpful - I’m afraid is has an effect you may not have considered. When others look at this thread and see a lot of responses from you, they may think that the problem is being addressed - and not take the time to contribute. So if you can’t offer something constructive, it may be better to stay silent.
The reason I hadn’t tried Ntrip is because there are numerous comments about the stable version of Mission Planner being buggy with Ntrip. But to clear up the concern you expressed, I loaded the beta and established an Ntrip connection. As the comments suggested - this worked and Mission Planner transmitted corrections properly using the Mission Planner beta.
Unfortunately, this didn’t change the result on my copter - as soon as I logged into the Ntrip Caster, the copter began experiencing GPS glitches.
This is the log file from that test.
What would be helpful would be to hear from anyone who’s successfully used an F9P receiver on their vehicle using RTK. Since the Here3-Pro is unavailable, I suspect few people have - so I expect any problems with that configuration haven’t yet come to surface.
I’m quite confident that my ground station is working properly. If you have time to look into the logs and see if anything is problematic, I’d appreciate that. While hunches have value, I’m hoping that someone knowledgeable about the firmware will have a few moments to look into this problem.
If it is a problem with firmware - the incidence of it will soon increase as more people install F9P receivers on their vehicles.
I looked at your logs. The GPS rapidly cycles between No Fix and an RTK fix state. There are no smoking guns in the parameters. It appears you are using a recent version of uBlox firmware.
Again, this suggests a problem with the RTK injection mechanism/comms.
I gave you the solution for NTRIP, which rules out a variable.
I answered your F9P question very succinctly.
I use all F9Ps and/or NTRIP + F9Ps, all with similar results whether using Mission Planner or direct radio injection of RTCM3.
To expand on that, I’ve used F9Ps on at least 9 vehicles, Copters and Rovers alike, and have troubleshot others’ problems with them countless times. I’ve helped discover and fix several bugs/anomalies in the uBlox driver and feature tested new driver options.
I also saw your post HOURS before responding. No one else did, so I offered what I could.
But apparently that’s just “trying” and “not constructive.”
My bad for missing your comment regarding your use of F9P’s.
If you could shed some light on your precise equipment and configuration, perhaps I can learn from your setup.
No doubt you believe you have helped many others. Perhaps you’ll be able to add this trouble report to your list of successes.
In the mean time I’m still hoping someone will offer some diagnostic assistance.
I’ve not noticed any changes to GPS/RTK handling in any of the changelogs to firmware since I’ve been involved. So my “hunch” is that as installing F9P’s on vehicles and ground stations is fairly new, some element of employing them may have something requiring a change.
I was sure hoping I had simply set a parameter incorrectly.
M8P fixed base direct USB connection to Mission Planner + MAVLink injection to M8P.
M8P fixed base direct USB connection to Mission Planner + MAVLink injection to F9P.
F9P fixed base direct USB connection to Mission Planner + MAVLink injection to F9P.
F9P fixed base UDP serial connection to Mission Planner + MAVLink injection F9P.
NTRIP internet connection to Mission Planner + MAVLInk injection to F9P.
F9P fixed base serial connection to SIK radio, SIK radio direct connection to F9P.
F9P fixed base serial connection to LoRa radio, LoRa radio direct connection to F9P.
F9P fixed base serial connection to ESP32, ESP32 direct connection to F9P, wifi protocol.
F9P fixed base serial connection to ESP32, ESP32 direct connection to F9P, ESP-Now protocol.
ArduPilot firmware versions have spanned Copter/Rover 4.0 through 4.3-dev.
The use of F9Ps isn’t as new as you might think, and the driver is fairly mature at this point, having been labeled “stable” through at least 3 major revisions. There were significant changes to it between 4.0 and 4.1. It has not seen much change in 4.2 and 4.3 because all testing indicates good/stable performance.
“No doubt you believe you have helped many others.”
All of your equipment citations are about fixed base. What are you using on your vehicles? Not sure if I mentioned but when using a Here3 on the vehicle, I did not have this problem.
Not sure if it matters - but it would be nice to know what brand and model receivers you’re using - not just the uBlox receiver type.
I know F9P’s have been around a while - but I’ve seen practically no mention of them in on-line conversations - and none about use in RTK operation.
With luck you’ll soon be able to add this to your list of successes.
What do you suggest I do to further the diagnosis at this point?
At the end of each line where I say “to F9P” my intention was to indicate that it is an F9P (or two) on the vehicle itself.
My M8Ps were HolyBro branded. All of my F9Ps are from ArduSimple (SimpleRTK2B and SimpleRTK2B+Heading).
Antennas have been either the magnetic uBlox branded patch antennas or (more frequently) survey grade antennas from ArduSimple (even on the Rovers themselves). I have not yet used helical antennas, though I have observed others’ success with them.
I suspect antennas are not your problem. The satellite count is quite high (upper 20s) and stable.
If you can take a laptop outside and connect it directly to a fixed base with a USB cable and to the autopilot with a USB cable, then run your RTCM3 via Mission Planner with those wired connections, we can eliminate the telemetry link as a possible cause. You won’t be able to fly that way, of course, but leaving it connected for 20-30 minutes in that configuration with LOG_DISARMED=1 may prove enlightening.
I’m also using the ArduSimple SimpleRTK2B as well on my base. I found a nice case for it on Thingiverse - works well as base receiver. I’m also using the SparkFun surveying antenna. (TOP106)
The SimpleRTK2B is connected via USB to my laptop I’m using as a base.
It appears the difference in our setups is that I’m using a Holybro receiver on my vehicle.
All of my logs have LOG_DISARMED=1. I think I mentioned that in my original post - my bad if I neglected to do so.
Please let me know if I can run additional tests and upload additional logs.
I installed the latest available firmware on the uBlox website on both my ArduSimple and Holybro F9P receivers.
In both cases I did not save and restore the parameters. (I saved them on the Holybro - but didn’t restore them)
So as I understand it, both receivers are using default parameters. And since the problem exists using Ntrip, only the Holybro (copter) receiver’s parameters might come into play.
As I understand uBlox parameters from using U-Center, changing a parameter puts them in volatile memory until a “save” is executed and the values are placed in non-volatile memory. I’ve found no mention of setting uBlox parameters for rover or base receivers to use them with ArduPilot or Mission Planner, I’ve always assumed that the ArduPilot firmware and Mission Planner will set any receiver parameters necessary at power up - and that they’ll not persist after power down.
It’s interesting about the Here3. There’s an option in Mission Planner to update the Here3 firmware - but I don’t believe it’s for the M8P receiver contained in the Here3 - rather for the Here3’s other processors. The update happens way too fast to be a uBlox receiver update. I suspect the only way to update the M8P firmware on the Here3 is by using CAN passthrough and U-Center. I tried to do that, but failed to get U-Center to connect to the receiver.
To answer your first post: since there are no issues when RTCM3 is not present, wiring is not a likely root cause.
The firmware and F9P configuration to date sounds very reasonable. Your understanding of u-Center and the GPS configuration-saving behavior is correct.
To be certain that the Copter’s GPS module is in its default state, it is advisable to connect to u-Center and use the screen shown below to “Revert to default configuration” (click Send) and then “Save current configuration” (click Send again).
I dug into your parameters again and compared them against a known working Copter configuration, which again turned up no clues.
Can you repeat the NTRIP test with a fresh boot of the autopilot, the fixed base physically disconnected from the GCS laptop, and a fresh instance of Mission Planner beta? (When I say “fresh instance,” I mean start MP and DO NOT connect it to any fixed base prior to activating NTRIP). I want to be sure we completely remove the fixed base from the equation.
On this log, the copter was started - and the Ntrip connection was established. I waited a couple of minutes before connecting mavlink to the copter.
The GPS glitches show up almost immediately. I went to LOITER mode and armed the copter. I spooled up the motors, but didn’t lift off. I disarmed. I was receiving regular GPS glitches the entire time.
Then I shut down the Ntrip server and shut down Mission Planner and the laptop. But I left the copter powered up.
It took about 30 to 60 seconds before the GPS glitches stopped. Oddly, the fast blinking green line on the copter receiver remained - as it does when it starts receiving RTK corrections.
I then put the copter in the air for a minute - no glitches.
After landing the copter’s receiver was still doing the fast blinking green.
Just a fast chime-in - I haven’t looked at the logs, but played a bit with F9Ps. Try using only GPS and Glonass, on both base and rover. And set the survey-in period for the base - you have to configure the base as base (check Drotek’s site for some info) - for twice the time it’s set at now.
May be a problem of 5Hz position while correcting for all 4 constellations or a “jumpy” base.
Disabling the unused serial port (de-selecting “GPS”) did not change the problem.
However, the direct wired connection you suggestion does seem to have solved the problem.
I’ve always had the baud rate of my SiK radio RFD900’s set to 57,600 - per the ArduPilot docs.
Now I’m wondering if that’s simply too slow to handle the the F9P corrections data stream.
I’m going to try increasing the SiK radio baud rate up to 230400 - which is the baud rate of the rover F9P according to the mavlink messages. That may be too fast to maintain a stable SiK radio connection on the RFD900’s - but it should tell us if we’re barking up the right tree.
Do you happen to know how fast a baud rate the RFD900’s should be able to handle?
If this turns out to be true, then it opens another whole can of worms. In fact - it appears to be open already. If upon entering RTK mode there’s a breach in connection with the corrections data stream, the GPS simply loses its lock, instead of reverting back to non-RTK mode - the vehicle will enter a EKF failsafe if the telemetry is lost because the GPS will lose it’s lock. This could be very dangerous should there be a break in telemetry - and might support the justification for a secondary GPS.
Having a go at baud rate was going to be my next suggestion. Recommend 115200 as a starting point. While I trust that @ThePara has successfully used lower data rates, that may be without the need to carry MAVLink2 on the same connection.
I have not seen any EKF failsafes, GPS glitches, or other anomalous behavior when RTCM3 injection is lost. The system simply reverts to DGPS and carries on. I use a script to detect undesirable fix types and pause navigation on some of my Rovers where RTK precision is more critical.
I suspect there is some sort of data corruption happening that is causing wildly poor behavior at present, and we just need to nail down a root cause.
I tried a test setting my SiK telemetry radios to 230400 baud. No change in the GPS glitches.
Here’s a log file of the results:
An EKF failsafe caused by loss of GPS is exactly what brought me to all of this. Per my previously noted uploaded log:
It seems that prior to copter 3.3 there was a FS_GPS parameter that specifically handled loss of GPS. Since then it’s been rolled into the EKF failsafe (FS_EKF_ACTION) - which has three options - “LAND”, “ALT-HOLD” or “LAND even in stabilize,”
I didn’t realize it but my copter triggered an EKF failsafe and had gone into “LAND”. Knowing I had lost the GPS, I switched the mode on my transmitter from LOITER to “ALT-HOLD” which enabled me to guide the copter down from about 100 meters overhead.
So if a loss of telemetry causes this GPS glitch problem, there’s a risk that under RTK operation that loss of telemetry could cause a EKF failsafe.
Cornel Fudulu’s comment suggests very high telemetry baud rates should be required - but I don’t know if he’s using an M8P or F9P.
I’m happy to test other baud rates - would would you suggest?
And of course I’m happy to try any other tests. I have the feeling we’re making progress.
We seem to have narrowed things to the telemetry connection. I have used SIK radios very successfully in the past. I usually set them to 115200, but I have one that was working quite well for RTCM3 injection at 57600, so baud rate is not the likely culprit.
Have a look at your telemetry radio config. Here’s mine, for comparison:
