New Thread - transmission buffer at 100% on SiK radios

jstroup1986 · July 30, 2022, 10:07pm

I thought it would be worth taking a step back and working get a better understanding of the SiK radios before going on troubleshooting the GPS glitches.

I’ll post about this in the hardware section, but I discovered that there are two hardware versions of the RFD900x. I couldn’t find the markings on the radios to tell which hardware version they were - but a kind contributor on facebook sent me this photo that shows where the markings are - in really small letters.

My original radios are Version 1, and my new radios are Version 2. I loaded the latest firmware on all of them, and discovered that Version 1 and Version 2 radios will not connect to each other. I’ve sent a message to RFDesign to see if this can be addressed.

I went back to Version 1 radios on my copters and base. I also set them up with the factory default settings.

Then without any Ntrip injection, I set up each of my copters, one at a time, to connect to my base for about 5 or 6 minutes. I wanted to get a baseline TxBuf for each. Here’s the result:

My Hexsoon EDU450 - the copter equipped with the F9P and the one that experienced GPS glitches when I started Ntrip injection for RTK. No Ntrip injection on this test.

The past logs on my Hexsoon TD650 showed far less TxBuf saturation - and that shows up in this apples to apples comparison - 5 or 6 minutes of connection with default SiK settings.

We’ve discussed ways to set up the radios so they can manage flow control better.

But perhaps the issue is more about why ArduPilot is sending so much data - even when the aircraft is just sitting idle in the workshop. Maybe there’s something in the BIN files of these two test that can uncover this. Both copters have very similar equipment and configurations. But obviously the TD650 has a less saturated TxBuf.

I don’t know if this is the right way to approach the problem. I’d welcome advice on a diagnostic strategy. I’ve uploaded the BIN files for these to sample runs to Dropbox.

Many thanks!

tridge · July 30, 2022, 10:16pm

The tlog is what we need to know what ArduPilot is sending.
On the v1 versus v2 RFD900x, I know RFDesign moved to a new MCU for v2, which means a new firmware. You can see the v1 and v2 firmware here:

jstroup1986 · August 1, 2022, 3:25pm

@tridge amd @Yuri_Rage and other followers -

I’ve spent this past weekend working this problem.

Andrew had proposed that a saturated SiK radio buffer might be the cause. Yuri reported that his rover works fine with F9P RTK and the TxBuf at about 100%.

I found a way to get the TxBuf to a lower level - and the GPS glitches still occur - and on this morning these GPS glitches led to two EKF lane switches and EKF fail-safes within about 15 minutes.

I’ve uploaded the BIN, T-Log and R-Log files of this test to dropbox:

Here are chart illustrating this mornings test run:

From this test this morning, I propose that the GPS glitch problem is not the result of TxBuf saturation. Of course if I’m wrong - I would be glad to shown otherwise.

From my I.T. experience decades ago, we’d likely run some sort of trace to find out what’s causing the GPS glitches. I have no idea what modern tools are available today - nor if any one of the ArduPilot firmware developer volunteers might be inclined to explore more deeply into this problem. But until it’s fixed, RTK navigation using my Holybro H-RTK F9P Rover Lite is impossible.

As info, here are the SiK settings that I used to reduce TxBuf saturation:

It occurred to me that by increasing Baud rate, the result was increasing the rate of bits from the hosts to the radio modems - which would have the effect of increasing saturation of the radio’s transmission buffer. I reasoned the way to reduce transmission buffer was to not increase the Baud rate, but improving the rate of data between the radio modems. Increasing the “Max Window” seems to have done that.

Again - as I’m not an expert on SiK radios - this logic may be wrong. However with this change the TxBuf is lower. If TxBuf is really is the cause of the GPS glitches, perhaps there are setting that can reduce TxBuf even further.

Also - the RTS/CTS wires do appear to be connected on both the base and rover SiK radios.

I really appreciate your hanging in there with me on this problem. I’d appreciate any suggestions on further investigation and analysis. Thank you!

jstroup1986 · August 1, 2022, 9:27pm

@tridge and @Yuri_Rage and other followers:

It just so happened that I had another Holybro receiver on order - the more expensive Holybro H-RTK F9P Helical - and it arrived today.

I installed it for comparison. It worked.

The key difference between this and the H-RTK F9P Rover Lite is that I upgraded the firmware on the Rover Lite - which re-set all the receiver’s parameters to defaults.

I saved the receivers parameters in a parameter file - and the thing to do now is to restore them to see if it makes any difference.

If it does - then perhaps ArduPilot should set the necessary parameters so that others who upgrade receiver firmware and forget to save or restore the parameters don’t suffer the same consequences that I have.

I’ll report what I find when I get the parameters restored.

Thank you!

jstroup1986 · August 4, 2022, 1:21pm

@tridge and @Yuri_Rage and other followers -

I appears I have solved this problem well enough - and also resolved the F9P RTK problem with GPS glitches.

The resolution to the SiK radio TxBuf saturation was to increase the Max Window parameter to 280. I still have a question out to RFDesigns for advice and clarification on maximizing throughput between radios - which should keep the transmission buffer at a minimum.

Here are the SiK radio settings I used:

The resolution to the GPS glitch issue during F9P RTK turns out NOT to be saturation of the transmission buffer - limiting the transmission of RTK corrections. Instead it was a firmware issue on the F9P receiver.

When I received the Holybro F9P receiver I uploaded the latest firmware - as it came with a firmware release that was 3 versions old. I saved the original parameters - but did not reload them.

I reverted back to the original F9P firmware version and reloaded the original parameters. That resolved the problem.

As an interesting aside - the parameters file that’s created when U-Center saves a receiver’s current parameters is tied to the version of firmware. You cannot upgrade the firmware, and reload the parameters from the older firmware version.

This creates a dependency on ArduPilot. Unless ArduPilot updates the uBlox parameters necessary for RTK, it’s dependent on a version of F9P firmware. It might work well enough to upgrade the firmware and to then change the F9P parameters manually with U-Center from the firmware defaults. But there’s no way to know what parameters should change - or to what values.

Unfortunately, the parameters file saved by U-Center is coded - you can’t read it to know what the settings were.

I’m going to close out my previous post on F9P RTK gps glitches by referencing this post - to avoid duplicating what I’ve entered here.

Here’s chart from a test flight - a simple climb in LOITER to 100 meters.

Yuri_Rage · August 4, 2022, 2:11pm

With the newest F9P firmware loaded, did you ever use the “Revert to default” button as I asked?

I suspect a more appropriate solution is to update the firmware, revert to default and save, and set GPS_AUTO_CONFIG=1, which creates an appropriate dependency on ArduPilot with a known starting point.

What you suggest in the post above could very likely break a moving baseline configuration, depending on how far back you have rolled back the firmware, and I do not recommend it as a solution.

jstroup1986 · August 4, 2022, 3:05pm

Yuri - No, I dropped the ball on that one.

The firmware is back to as it came from the factory - so I doubt that the older firmware version is causing difficulties. If it were a problem, there would have been numerous issues raised about it - and I’ve seen none.

In fact the only place I’ve specifically seen instruction in ArduPilot to upgrade receiver firmware is on the CubePilot docs on doing a firmware upgrade to their earlier M8P receiver sold in their RTK base product. In those instructions you have to open the receiver shell to find a USB connection - and use that to update it with U-Center. Interestingly, I’ve taken a Here3 apart and there is no USB connection in side - just a JST SH 1.0 connector without documentation. I think the intention is that CAN Passthrough be used to update Here3 firmware - but even after dialog with Phillip Rowse, no one can demonstrate how that works. There is new and old documentation - none of it worked for me.

The GPS_AUTO_CONFIG parameter is only or serial connected GPS - so I’m guessing it only deals with things pertaining to the serial connection - an not any of the other uBlox parameters. Mine is set to “1” - and I didn’t change it - so I’m guessing that it’s the default.

The serial port I have the F9P connected to is set at 115200, but there is a MAVLINK message on startup that reports resetting it to 460800. I’m guessing that’s the result of the auto config.

My understanding of the U-Center “Revert to default” function is that if after making changes, it resets the receiver to the defaults. I also understand that when loading new firmware, the default parameters for that firmware version is set. That’s the reason for the ability to save current parameters to a file. Do you have different info about that? If so - I’d like to see a reference.

What is unknown is if Holybro changes any parameters when they ship the F9P. Because the parameter file is coded - there’s no way to easily see what the existing parameters are - and the doing the “revert to default” - and then comparing them to see if Holybro made changes.

What is also unknown is what parameters ArduPilot expects. Without any statement about it, ArduPilot either expect the parameters will work as the product comes from the factory - or ArduPilot takes responsibility to set the parameters properly. The only place that suggest that ArduPilot takes responsibility for receiver parameters is on the RTK/GPS INJECT page - where there are check boxes for “UBlox M9P/F9P autoconfig” and “M8P fw 130+/F9P”. I select both of those.

There’s nothing corresponding to those check box options for a rover using those receivers.

There’s certainly a lot to know about all this - and I appreciate your help learning.

Yuri_Rage · August 4, 2022, 3:10pm

See firmware update recs here:

U-Blox F9P Firmware Update — Copter documentation (ardupilot.org)

See note regarding old firmware and moving baseline here:

GPS for Yaw (aka Moving Baseline) — Rover documentation (ardupilot.org)

ArduPilot expects a default config. There are simply too many parameters to hard code into the ArduPilot autoconfig routine, so it only changes the ones that need updating from default.

I have DEFINITELY observed a “revert to default” procedure fix many issues with ArduPilot connected F9Ps, because, as you say, there is no way to know what’s on there from the start.

jstroup1986 · August 4, 2022, 3:28pm

Interesting - I remember reading this.

Fortunately, I’m not doing moving baseline - although I might in the future.

The firmware version that came with the Holybro is HW Version 112. That’s what I’m using for now.

I tried loading version 113, but it wouldn’t accept the parameters I had saved from the original. So I didn’t test it.

It is possible that version 113 would work with the firmware defaults - but I haven’t tried it.

I can easily see how the “revert to defaults” could save someone’s bacon from time to time. But on loading fresh software, the parameters are all supposed to be at the defaults for that version - so that’s why it didn’t occur to me to try it.

It’s difficult for me to try options like this because of the connector. The Holybro receiver I’m using has the 10-pin connector - and I have to move pins over to an 8-pin connector to attach to the CubePilog carrier board. The Holybro came with a really nice FDTI device to convert the 10-pin GH to USB. Using their converter requires re-pinning to a 10-Pin connector and back. I’m concerned on the stress that puts on those wire and connectors. I’ve ordered some 10-pin and 8-pin female GH connectors - when I get them I’ll make an adapter which will make testing a whole lot simpler.

Yuri_Rage · August 4, 2022, 3:31pm

What you have done seems to be working for your specific use case. I completely understand the frustration that uCenter and fragile connectors bring.

However, F9P firmware reversion should not be the recommended solution for others.

jstroup1986 · August 4, 2022, 3:46pm

I think we’ve had a mis-communication.

I’m not advocating a firmware reversion. I upgraded the firmware from the factory installation - and simply went back to what was on the factory installation.

Yuri_Rage · August 4, 2022, 4:02pm

1.12 is the one known to have issues with moving baseline configs, which are a very common use case for F9P receivers and ArduPilot.

1.32 is current as of this post, and I can confirm good performance with ArduPilot.

jstroup1986 · August 4, 2022, 4:14pm

Interesting that you’ve had success with 1.32. That’s the version that caused the GPS Glitches for me.

But I didn’t click on “revert to defaults” - because I assumed that when loading the firmware it automatically set the defaults. I might have to ask about this on the U-Blox support portal.

jstroup1986 · August 4, 2022, 4:23pm

https://portal.u-blox.com/s/question/0D52p0000CeTcGbCQK/f9p-default-parameters-when-upgrading-firmware

tridge · August 4, 2022, 8:55pm

i’m glad you’ve resolved it and thanks for posting your results, although I’m curious what the difference in parameters is that matters for this case.
u-blox receivers have a huge number of parameters, and ArduPilot only overrides a small subset of those. It sounds like there is a key parameter for good RTK performance that we are not overriding.

xfacta · August 4, 2022, 10:28pm

Also the RFD radios work well at higher baud rates if that helps.
I dont do long range myself, but I’ve seen the same settings on other sets of radios still achieve very long range. Works well for .bin log downloads too.

Baud 115
Air Speed 200
Max Window (ms) 80

Also you have to use 115k baud in MissionPlanner, instead of 56k of course.
And you have to set your telem radio serial port to 115k, such as SERIAL1_BAUD,115

jstroup1986 · August 5, 2022, 1:50am

Thank you Shawn for getting on board with this discussion.

What started out as an exploration into why I was getting GPS glitches when attempting F9P RTK turned into an exploration into telemetry throughput. This was the result of @tridge noticing that the Transmission Buffer reported in the BIN file was saturated - 100% for my tests.

So I turned my attention to telemetry throughput before looking into the GPS glitch problem.

As I see it, there are three possible choke points in the telemetry link.

The base (Mission Planner) to the base telemetry radio
Between the base and rover telemetry radios
The rover (ArduCopter) and the rover telemetry radio

As I understand it, the saturated Transmission Buffer suggests that the second case, the link between the two telemetry radios was a choke point. It might not be the only one - but it’s the one we have a statistic about.

There are two ways to reduce the saturation between the telemetry radios:

increase the throughput between the two telemetry radios, and
reduce the amount of data being served to the telemetry radios by the base or rover. (Mission Planner or ArduCopter.

The BAUD parameter on the telemetry radios and associated serial ports on the base and rover control the rate that data is served to the telemetry radios. Increasing the BAUD rate has the potential of making the link between the telemetry radios the choke point. That’s why I’ve kept the BAUD rates of both the base and rover at 57600.

There’s very little information in the RFDesigns documentation about how to increase the bandwidth between the radios. And there is the issue of how the data speed between the radios may affect transmission distance.

The two parameters I think can speed the data link between the radios are Air-Data-Speed and Max-Window. I’ve posted a question to RFDesigns for more information about how to use these parameters - or others I’m unaware of - to improve data throughput between the radios.

In my exploration, I found that by keeping the BAUD rates at 57600, not changing the Air-Data-Speed, and increasing the Max-Window to “280” produced a measurable and significant reduction in Transmission Buffer saturation.

I tried increasing Air-Data-Speed and did not get improvement.

There is no statistic (I’m aware of) that reports if there is saturation of the bandwidth between the base or the rover and their radios. It would be nice to have those statistics.

As it happens, @Yuri_Rage reports that his F9P RTK implementation works without GPS glitches when his reported TxBuf is 100%. So all this may be an unnecessary exercise - but I’m compelled to do due diligence on the telemetry link.

I discovered that if the GPS experiences too many glitches, ArduCopter will do an EKF lane switch and possibly an EKF Fail-Safe. This is a scary thing to consider. There may be some situation where there is simply no way to prevent this - but I’d like to make sure that the telemetry link is not a potential source of the problem.

What might be a nice fallback is if ArduCopter sees an GPS glitch during RTK, that it reacts by severing the RTK, and attempting a recovery with non-RTK GPS. If that succeed, then at least an RTL is possible. I read that since FS_GPS was eliminated after Copter 3.3 and was rolled into the EKF failsafe, maybe this needs to be revisited.

In the mean time, I’ve learned that upgrading firmware on a uBlox receiver may be a little more complicated than just loading the new firmware. I’m investigating that with the uBlox support forum.

https://portal.u-blox.com/s/feed/0D52p0000CeTcGbCQK

xfacta · August 5, 2022, 3:48am

I suspect their doco just lists the bullet-proof specs, rather than being the full capabilities. For example I know the radios have been tested in ridiculous ways to destruction, but only realistic specs are provided where RFD would probably guarantee operation.
I’m aware of other people using those higher rates too, as well as myself. It works well.
EDIT: I don’t do RTK though
I’ve seen the RFD900X’s communicate reliably over 16km of urbane sprawl at ground level with that increased baud rate. You can theoretically use these things for remote support and I know that’s happened too!
The original 3DR style radios are not really capable of increased baud rate for a number of reasons.

Increasing the Air rate alone doesnt do much for you - that’s the underlying rate that the radios comm with each other, and may only affect something like RC control using the PWM or SBUS passthrough feature.
You have to also increase the baud rate and decrease the window to see any affect at ordinary serial (mavlink) level.

FYI, I’m not an RFD sales person, only a loose association like anyone that would purchase their products.

Yuri_Rage · August 5, 2022, 7:19am

I suspect there are two distinct and fairly unrelated rabbit holes here.

#1 RFD900 settings optimized for range OR optimized for throughput (mutually exclusive)

#2 F9P settings compatible with ArduPilot RTK injection

It seems that as long as #1 is reliable for connectivity, RTCM3 is a non-issue.

And as long as you revert to defaults prior to connecting an F9P to an autopilot, #2 is a non-issue.

Feel free to dive as deep as you like into either, but neither of these appear to be systemic issues.

jstroup1986 · August 5, 2022, 6:11pm

@tridge @Yuri_Rage @xfacta

As long as people remember that simply increasing the baud rate parameters alone won’t increase the bandwidth of the telemetry radio - everything should work just fine.

At baud of 57600, Air-Data-Speed must be “64” or higher. But there’s no benefit to faster Air-Data-Speed at baud of 57600 - in fact range will go down if it’s set to higher.

At baud of 57600 - that’s the choke point. It equates to about 7000 characters per second. (without ECC, resend, etc.)

In RTK operation, there are two components of data going across the telemetry link. 1) the telemetry, 2) the RTK corrections. The telemetry can be throttled by the Mission Planner config screen:

I don’t know how much data is contained in the RTK corrections. I expect it has to do with the HZ set on the base GNSS receiver. With Ntrip - I don’t know.

Setting BAUD up to 115200 ought to provide more headroom - as long as Air-Data-Speed is also increased appropriately.

It sure would be nice to know how much bandwidth of the telemetry link is required - because increasing Air-Data-Speed reduces RFD900x range. Greater range equates to greater reliability - we all like that.

I only found one reference to any of this the the RFDesigns docs - and oddly, there is a similarly named doc that doesn’t have this note. (see the note in the red box)