SERIALx baud rate for uBlox GPS modules with GPS_AUTO_CONFIG enabled is overridden. You should see a message on boot telling you that it’s detected at 230400 (I think we may have been typing at the same time - you confirmed this above).
EDIT: you just said you had a spare set of radios. Confirm the problem persists with the new radios?
Yes - problem is the same with new radios. Even with the new settings. They’re identical models to what I’ve been using. Thanks!
Also, while not helpful to the problem at hand, it’s important to note that the air data rate is independent of the wired serial rate. You could set differing baud rates on either end and still achieve successful data transmission so long as the wired connections were set to the respective, correct rates on the GCS and autopilot. Any one of these can create a bottleneck when bandwidth is exceeded.
RTCM3 injection should not exceed the available bandwidth for your configuration as it stands, so I’m a bit stymied for the moment. Hoping @tridge has some more ideas. The radios you’re using are typically known to be of high quality and well worth using.
Have you reduced the MAVLink message data rates as he suggested?
Good catch - I missed that one. And I’m not actually sure how to set Mission Planner to lower the telemetry streaming rate - to 3Hz or below - on all streams.
By copy to @tridge - can you point me in the right direction to make those reductions. (I tried a quick google to figure it out - but didn’t find anything that seemed appropriate.)
Many Thanks!
Maybe this is how to set the Mission Planner streaming rates. All mine are at “2” except Attitude, which is set to “4”.
Yes, that’s the correct section for global changes. You might also check the SR1_* or SR2_* parameters in the complete list for more granular control (I don’t recall which port you’re using for the radio, but I think it’s one of the two). ADS-B, in particular, could possibly be a bandwidth hog.
OK - I set all the Mission Planner telemetry rates to “1” - and still got the GPS glitches - but no red warnings about “Unhealthy GPS Signal”.
And ADSB is not active on my copter - I’m not even using an ADSB carrier board.
So then I went one step further and set just two items at telemetry rates of “1”.
Testing that, I still got GPS glitches. Perhaps a bit less severe - bot got them way to often for any level of comfort.
I’m sorry our tests today weren’t more fruitful. Thank for hanging in there with me!
One thing that occurred to me - I’m using YAAPU passthrough over CRSF - I get YAAPU telemetry on my transmitter. I know this doesn’t have anything to do with the SiK radios - but it’s just another piece of the telemetry puzzle.
Passthrough telemetry shouldn’t affect RTCM3 injects. I use it on most of my vehicles with zero detriment (all H7 processors, but varying exact hardware).
What autopilot/carrier board are you using? Perhaps there’s an impact there…?
I’ll submit that ANY GPS glitches that seemingly occur due to some particular configuration (as opposed to actual/external GPS anomalies) are 100% unacceptable.
BTW Andrew - the graph you have above showing RAD.TxBuf - that looks like a MavExplorer graph - but I don’t seem to have it on my copy of MavExplorer. Can you please tell me where you got it?
Tridge wrote most of (if not all of) the code for MAVExplorer, so I’ll defer to him for graphing there. However, here is the same graph displayed via Mission Planner’s log viewer for your most recent log:
Interestingly, I connected a little testbed Rover that uses mRo radios with SIK firmware tonight, using NTRIP RTCM3 via Mission Planner MAVLink injection and graphed the same parameter with this result:
It shows a completely saturated transmit buffer (at 115200 on both ends), yet I have had no issues at all with it during any phase of navigation. It happily zips around my yard at a surprisingly high rate of speed with RTK precision.
I can try the same test on my largest Copter (with matched firmware to yours) at a later time. It runs a Cube Orange with SIK radios whose settings I showed earlier in the thread and an M8P GPS. I do not think the exact GPS model is to blame here, so such a test should be valid (or at least a reasonable data point).
My preferred autopilot is the Cube Orange, and I have always used the awkwardly configured ADS-B carrier board simply due to availability (I don’t need ADS-B on any of my builds, even the airborne models, though I do enable it on anything non-Rover). I have achieved similar success with the entire range of Matek H743 boards without issue, and I have even gotten reliable RTK Fixed performance from an extremely cheap Pixhawk knock-off with GPS hardware far beyond its class.
This particular issue remains a mystery to me, and I am eager to find the solution!
OK - Thanks to Yuri’s pointing out that I can graph RAD.TxBuf in Mission Planner, I dug into the history of this on both this copter testing RTK (Hexsoon EDU450) and my other copter not involved in these tests. (Hexsoon TD650)
I copied the complete contents of the log directories from both copters SD cards so I could easily compare their log files. I’ve put them up on Dropbox here:
Both copters are very similar. Both use TBS Blacksheep Crossfire RC radios running Yaapu. Both have RFD900x telemetry radios. The EDU450 (the RTK subject) uses a CubePilot mini carrier board, the TD650 uses a CubePilot ADSB standard carrier board. The EDU450 uses the little “Power Brick Mini” to sense current and power the Cube - the TD650 uses a MAUCH current sensor and BEC to power the Cube.
The EDU450 has a FPV camera and VTX - installed about a year ago - but neither the camera nor the VTX touch the autopilot at all.
Looking back through these logs, the TD650 does not experience the TxBuf saturation that we’re seeing on the RTK test on the EDU450. Here’s a typical graph from the last TD650 flight - a map survey flight: (It’s the last TD650 log file in the uploaded directory.)
Going through the logs on the EDU450 - they all show high TxBuf saturation - going back to the beginning. I installed a new RDF900x on the EDU450 yesterday - it did not change the TxBuf saturation.
So while we’re only assuming at this point that the TxBuf saturation on the EDU450 is causing the GPS glitches once Ntrip injection begins, there is something obviously different in these two copters.
I’m a bit curious if maybe there’s something different in the Cubes themselves - or perhaps how they are powered differently.
When I fly the TD650 I frequently have QGC as a telemetry base - I just use it for situational awareness and redundancy. But I don’t always use the base - as I have Yaapu that provides telemetry to my RC control radio. Fewer of the EDU450 flights are done with the connection to QGC as a telemetry base - must most of the longer flights do use it.
One last note - on a few of the logs there is not RAD selection on the Mission Planner log graphs. I’m not sure why this would be the case.
il post this here from your facebook post
TX buff is on the telemetry radio not the flight controller, it tells the flight controller the status of the radios transmission buffer so it can dynamically control the packet rate in order not to overload the radio link and start dropping packets.
This is how you can have a serial rate much higher than the transmission rate without it dropping packets since the radio tells the flight controller how much it can handle in real time.
this stat is only available on some specific radios that are mavlink compatible since they need to interact with the flight controller, the most common are the old 3dr radios and the rfd900.
radios that provide a transparent serial link dont need this as serial speed = link speed.
its totally normal for the buffer to be 80-90% full since it would just speed up the packet rate if it start to drop to keep it there.
check your radio settings are identical, copy the settings from the working copter to the one with issues. its possible its running so slow that its having to slow the link to the point the gps injection stops working.
you need to check the status while its working look for rssi, noise remnoise, remrssi, to see if there are any unusual numbers by comparing between the pair of machines and pairs of radios. you could have a radio with a noisy regulator or something jamming the radio causing it to slow down. I used to see it with original 3dr radios that had a usb chip that would jam 433 radios.
the txbuf message is sent as a mavlink radio status packet.
Thank you GeoMuir - I appreciate your input.
As I mentioned earlier in this thread, I replaced both the base and rover radios yesterday - same model - but appear to have newer firmware. No change in the problem - so I’m guessing that eliminates the noisy radio problem.
As I also mentioned - it’s uncertain that the saturated transmission buffer is the cause of the GPS glitches - it was just something that was proposed. As Yuri noted also above in this thread, he has an F9P RTK rover running fine with a saturated transmission buffer.
I would like to know why one of my copters has a saturated transmission buffer and the other does not. The radio setup is identical. I have only one base radio - so I set all my “rovers” up to match it. You’ll also notice from the thread above that I shut down almost all the telemetry threads on Mission Planner - so it’s a mystery at this point as to what’s going out over the telemetry radios.
As I asked on your facebook response to my post, I’d also like to know how is it that the ArduPilot firmware knows and logs the radio’s transmission buffer percentage.
By the way - before attempting RTK I typically use my telemetry radios to link to Qgroundcontrol running on an Android tablet - just for situational awareness - and as a backup. I use Yaapu, so my primary telemetry information is displayed on my control radio. Anyway - I’ve never had issues with the telemetry link to either of my copters with the Qgroundcontrol ground station.
If you have time, I’d appreciate it if you’d read through this entire thread, and maybe look at some of the logs. I’ve uploaded the entire SD card contents of both my copter’s logs up on Dropbox.
Thank you for staying in the loop. As this problem came to light because the GPS glitches were so severe that it caused a EKF Failsafe, I’m guessing lots of people would benefit from knowing what’s happening here.
Regarding the radios TxBuf, the significance of the buffer being full depends on whether the GCS is using hardware flow control to the radio.
The radio has (or should have!) 6 wires that go to the GCS. They are RX, TX, GND, 5V, RTS and CTS. The RTS and CTS pins are “hardware flow control”.
In the RFD900x the radio sets the CTS pin high when there are less than 17 bytes free in the buffer and sets it back to 0 when there are more than 34 bytes free. If the GCS software (presumably mission planner) sees this as high then it should stop writing to the serial port. This matters as if it keeps writing then the data will be lost as the radio has nowhere to put it.
Note that there is a RTSCTS option in the radio. That controls whether the radio looks at the RTS pin to determine if it is able to write bytes to the GCS. The radio sets the CTS pin to 1 when the TxBuf is full regardless of the setting of RTSCTS.
The reason I mention all this is that it is possible that either MissionPlanner is not correctly looking at the CTS pin, or the CTS pin isn’t being carried through whatever USB serial adapter you are using. That would cause the effect you’re seeing, where packets being sent to the aircraft are corrupted.
The way you check these things is with a logic analyser looking at the RX and CTS pins on the radio. If the RX pin shows serial data being sent (ie. data from GCS → radio) while CTS is high then you know that hardware flow control is not working.
Cheers, Tridge
Thanks Andrew -
On the Mission Planner screen for setting SiK radio parameters there is a check box “RTS CTS” on both radios. Both these check boxes are blank.
Should I check them?
My physical connections to the carrier board on the copter and to the PC running Mission Planner are ordinary, common, and not configurable to the best of my knowledge.
On the copter, I use the cable/harness that comes with the SiK radios designed to connect to a carrier board Telem port.
On the ground station, I have the SiK radio mounted in a BASK Aerospace “AeroLink Base” that holds the RFD900’s and has it’s own FDTI between the radio’s serial connections an an external USB connector.
Does any of this sound suspicious?
Many thanks and regards!
the RTSCTS options in the radio don’t matter in this case, as that is what stops the radio from sending data when the recipient isn’t ready, but the issue you are presumably seeing is the GCS sending data when the radio isn’t ready.
I’m not familiar with the BASK AeroLink Base. I would hope it connects the RTS and CTS pins, but it is possible they saved 2 wires and left them out. You’d need to open it up to check.
I will check.
In the mean time, I can also try using the FDTI cable provided by RFDesign.
I have resolved this problem.
Description of the resolution is at the end of the thread on this post: New Thread - transmission buffer at 100% on SiK radios
The problem turned out to be with the firmware I’d upgraded to - and perhaps it’s parameters. By reverting to the firmware that came on the F9P receiver (which was three versions old) the problem was resolved.
I also found a way to reduce TxBuf - and I’m still researching that further. But I’m not sure that issue influenced the problem.
Here’s a chart from a successful test.
I owe @Yuri_Rage an acknowledgement.
Sorting through the F9P RTK issues, Yuri suggested doing the U-Center function that resets the firmware parameters to their defaults. He said that had cleared up problems for him in the past.
I thought I had read that when loading new firmware, the old parameters would be cleared and re-loaded with the defaults for that version of firmware. I upgraded my Holybro F9P Helix receiver from 1.30 to 1.32 and found that even after the upgrade I needed to do the parameter reset to their defaults in U-Center. The F9P receiver wouldn’t function properly in RTK without it. Who knows what else it wouldn’t do.
I think this is an important point because I don’t recall reading anywhere about resetting parameters to their defaults when upgrading u-Blox firmware. Thank you Yuri for suggesting it…
I didn’t latch on to your comment when it came through - My apologies.
While this problem had a couple of dimensions - you were spot on regarding one of them.
On advice from the u-Blox support portal, I configured the receiver (via ArduPilot parameters) to only use GPS and Glonass - as you suggested in the first place. I was also advised to disable SBAS - which I did.
The results were spectacular. I was able to set the refresh rate to 10Hz, and I get a RTK-FIXED lock almost immediately on power up.
