Rover-4.1.0-rc1 released for testing

Rover-4.1.0-rc1 has been released for testing and can be downloaded using MP or QGC’s beta firmware link. The changes vs -beta7 are in the release notes and copied below

Changes from 4.1.0-beta7

  1. Enhancements
    a) DCM fallback easier on boats
    b) Flywoo F745 Goku Nano support
    c) MatekF765-SE support
  2. Bug Fixes
    a) ChibiOS scheduling slip workaround to avoid occasional 60ms delays found on MatekH7
    b) EKF2 divide-by-zero protection when using optical flow (issue only found in simulator)
    c) External AHRS (e.g. VectorNav driver) init fix
    d) KakuteF4Mini SBUS fix
    e) Pixhawk4 blue and red LEDs swapped

Thanks again for all your help with the beta testing!

As @Yuri_Rage mentioned over on this thread Rover-4.1.0 is on the cusp of being released officially. The issue list is not completely clear but many of these items are not blockers to the release.

The “PRX missing from 1MB boards” issue is perhaps the most serious but I’m tempted to resolve this in a point release which probably won’t be that far off because we are going to try and keep Copter and Rover in sync from now on.

My feeling is that 4.1.0 is a large improvement over 4.0 and we could release it any time. What do you all thing?

6 Likes

I completely agree that 4.1.0 is a huge step forward, and it’s quite stable.

I have one concern and one request for RC1:

Concern:
It appears that the BendyRuler algorithm has changed quite a bit over the past couple of beta releases, and between beta6 and beta7, it got very “touchy.” By that, I mean the behavior can be somewhat unpredictable if a waypoint is placed close to a fence. I endeavor to always put my waypoints at least the avoidance margin away from any exclusion fence, but lately it seems I can’t get them far enough away to keep BendyRuler placated (in other words, to keep the Rover from wandering too far from the expected path).

Request:
If at all possible, please take another look at the triggers for “Unhealthy GPS Signal” messages, particularly in a moving base configuration. The issues list pretty clearly spells out the situations where those messages become frequent, and they appear to be unnecessary - occurring while the fix type is stable, positional accuracy is tight, and EKF innovations are small.

1 Like

Similar to @Yuri_Rage, I also noticed after installing beta7 that BendyRuler caused wide swings around an exclusion polygon. I had changed a couple of parameters prior to the run with beta7 as recommended by @rishabsingh3003, so that could be some of it. I will try to test a little more tomorrow and provide feedback.

But I have had no issues, other than the 2 that Yuri mentioned, with beta7, and I have run it a pretty good bit.

1 Like

There was one small change to BendyRuler over the past month, but apart from that there have been no significant changes (especially to Rover side) since last year… I’ll talk to Randy about this and try and reproduce.

1 Like

Is Scurve destined for an update after 4.1 release?

@oaamaas,

Yes, s-curves will be one of my top priorities after 4.1 goes out. I can’t say yet whether that will end up being a 4.1.x release or 4.2.x release but it will be available for testing one way or the other.

2 Likes

I ran RC1 for a good long while today, and here are my observations:

I don’t think BendyRuler is as much of a concern as I made it out to be in my previous post here. Rather, I think the improvement(s) since beta5(?) have made the algorithm a little more faithful to the FENCE_MARGIN parameter. I previously had the margin set to 1m, and I was getting away with putting waypoints VERY close to, if not within that margin while keeping them outside the actual fence confines. Reducing FENCE_MARGIN to 0m with the newer releases seems to result in nearly identical behavior as I was seeing with a 1m margin before. I hope that’s a clear enough explanation of my previous concern and resulting fix. A possible recommendation for the Wiki/documentation pages: Ensure that waypoints fall outside geofence confines + any avoidance margin, and use care when placing sharp turns in the vicinity of geofence boundaries.

With RC1, I’m actually seeing relatively few “Unhealthy GPS Signal” messages, but I am seeing quite a few “No Fix” messages (a few milliseconds each) with GPS_RATE_MS*=200. Take a look at the timestamped messages below where I’m running a script that reports changes in GPS1 status. There are 26 satellites in view with good RTCM3 messaging. This “No Fix” to “Good Fix” issue has been present for a while, and it seems it’s triggering fewer “Unhealthy GPS Signal” messages now, but I’m not sure why the fix status is changing (or reporting changed) rapidly like that under what should be good, if not ideal conditions.

The mower is running a pretty long waypoint mission right now, but I can run a shorter series of waypoints later and grab that log to show the issue if you like.

9/3/2021 15:35:53 : Reached waypoint #78
9/3/2021 15:35:50 : GPS 1: RTK Fixed
9/3/2021 15:35:50 : GPS 1: No Fix
9/3/2021 15:35:49 : GPS 1: RTK Fixed
9/3/2021 15:35:48 : GPS 1: No Fix
9/3/2021 15:35:48 : GPS 1: RTK Fixed
9/3/2021 15:35:48 : GPS 1: No Fix
9/3/2021 15:35:45 : Mission: 78 WP
9/3/2021 15:35:45 : Reached waypoint #77
9/3/2021 15:35:40 : Mission: 77 WP
9/3/2021 15:35:40 : Reached waypoint #76
9/3/2021 15:35:36 : GPS 1: RTK Fixed
9/3/2021 15:35:36 : GPS 1: No Fix
9/3/2021 15:35:31 : GPS 1: RTK Fixed
9/3/2021 15:35:31 : GPS 1: No Fix
9/3/2021 15:35:28 : Mission: 76 WP
9/3/2021 15:35:28 : Reached waypoint #75
9/3/2021 15:35:24 : GPS 1: RTK Fixed
9/3/2021 15:35:24 : GPS 1: No Fix
9/3/2021 15:35:22 : GPS 1: RTK Fixed
9/3/2021 15:35:22 : GPS 1: No Fix
9/3/2021 15:35:17 : GPS 1: RTK Fixed
9/3/2021 15:35:17 : GPS 1: No Fix
9/3/2021 15:35:09 : GPS 1: RTK Fixed
9/3/2021 15:35:08 : GPS 1: No Fix
9/3/2021 15:35:08 : GPS 1: RTK Fixed
9/3/2021 15:35:08 : GPS 1: No Fix
9/3/2021 15:35:07 : Mission: 75 WP
9/3/2021 15:35:07 : Reached waypoint #74
9/3/2021 15:35:05 : GPS 1: RTK Fixed
9/3/2021 15:35:05 : GPS 1: No Fix
9/3/2021 15:35:04 : GPS 1: RTK Fixed
9/3/2021 15:35:03 : GPS 1: No Fix
9/3/2021 15:35:03 : GPS 1: RTK Fixed
9/3/2021 15:35:03 : GPS 1: No Fix
9/3/2021 15:35:02 : GPS 1: RTK Fixed
9/3/2021 15:35:02 : GPS 1: No Fix
9/3/2021 15:34:52 : GPS 1: RTK Fixed
9/3/2021 15:34:52 : GPS 1: No Fix
9/3/2021 15:34:51 : Mission: 74 WP
9/3/2021 15:34:51 : Reached waypoint #73
9/3/2021 15:34:46 : GPS 1: RTK Fixed
9/3/2021 15:34:46 : GPS 1: No Fix
9/3/2021 15:34:46 : GPS 1: RTK Fixed
9/3/2021 15:34:45 : GPS 1: No Fix
9/3/2021 15:34:41 : Mission: 73 WP
9/3/2021 15:34:41 : Reached waypoint #72
9/3/2021 15:34:39 : GPS 1: RTK Fixed
9/3/2021 15:34:39 : GPS 1: No Fix
9/3/2021 15:34:37 : GPS 1: RTK Fixed
9/3/2021 15:34:37 : GPS 1: No Fix
9/3/2021 15:34:35 : Mission: 72 WP
9/3/2021 15:34:35 : Reached waypoint #71
9/3/2021 15:34:35 : GPS 1: RTK Fixed
9/3/2021 15:34:35 : GPS 1: No Fix
9/3/2021 15:34:18 : Mission: 71 WP
9/3/2021 15:34:18 : Reached waypoint #70
9/3/2021 15:34:11 : GPS 1: RTK Fixed
9/3/2021 15:34:11 : GPS 1: No Fix
9/3/2021 15:34:11 : GPS 1: RTK Fixed
9/3/2021 15:34:11 : GPS 1: No Fix
9/3/2021 15:34:10 : Mission: 70 WP
2 Likes

I ran a 4 hour mission with 583 waypoints this afternoon with 4.1.0-rc1. I had a script written by Yuri running which reports the GPS status changes also. I saw a few short drops from RTK Fixed to RTK Float or DGPS Fix a few times, but only a few, as in maybe 10 over the 4 hour period. I don’t think I ever saw it drop to No Fix. Does that “No Fix” really mean no GPS fix or just no RTK Fixed? Some of my dropouts were legitimate caused by tree cover and those lasted 10 or 20 seconds, but most were the very short blips. I wonder if short drops from RTK Fixed have always occurred but are so short that Mission Planner either doesn’t see them or doesn’t have time to report them, whereas the Lua script catches them. I guess comparing an old .bin log to a newer one would tell.

1 Like

The fact that Kenny is using very similar hardware but not seeing the same results makes me wonder if I have a configuration or hardware problem. I will investigate a bit.

1 Like

I did some digging today, and I could not solve the intermittent “No Fix” issue on my own. I double checked all settings per the documentation. I did still have SERIALx_OPTIONS=768 (no DMA) from our previous testing of a different configuration, so I reset those to 0, but that did not help. I’m using GPS_AUTO_CONFIG=1 and GPS_DRV_OPTIONS=1.

I’m seeing “RTK Fixed” -> “No Fix” -> “RTK Fixed” sometimes 2-3 times in less than a second followed by “RTK Fixed” for a longer period of time.

I’ve been seeing this behavior ever since I followed @Tridge’s explicit instructions to set GPS_RATE_MS*=200. I didn’t see these dropouts with the faster 10Hz rate, but we definitively proved that the 5Hz rate is better for yaw performance (as expected!).

I’m happy to provide a .bin log showing the issue if you like.

EDIT:
Some further testing this morning showed the same issue when telemetry and RTCM3 messaging was connected directly through the Mission Planner laptop rather than through my telemetry/fixed base wifi repeater (which is encouraging about my hardware setup…but discouraging regarding the “No Fix” situation).

FURTHER EDIT:
Supporting log file can be downloaded here.

GPS[0].Status showing rapid changes from No Fix to RTK Fixed:

1 Like

I hope to get out and do a little more exploration of the “No Fix” issue today. I made a custom RC1-based firmware build last night to force uBlox GPS modules to 460800 baud instead of 230400 to see if that might have an effect.

I will also try direct RTCM3 injection vs MAVLink injection, but that may require some hardware I don’t have on hand, so it may take a few days for that to happen.

EDIT:
No change using 460800 baud - still seeing blips of “No Fix” just like before.

I also tried changing the GPS health check to 245ms for both base and rover uBlox GPS types with no improvement there, either (I didn’t suspect that would help…and it didn’t).

However, if I disable MAVLink RTCM3 injection from the fixed base, GPS1’s fix type stays steady at “3D Fix” with no drops to “No Fix.” So I think I can confirm that the issue is related to the heavy message burden/bandwidth required by RTCM3 through the flight controller. I wonder if the real problem is the 57600 baud rate that I use for SIK radio telemetry?

I should be able to test direct RTCM3 injection later today or tomorrow.

1 Like

I might have found an issue with Rover-4.1.0-rc1 running on Matek H743-Wing failing to accept PPM inputs.

If I don’t change my setup at all other than load the current stable version of Copter, I get PPM input showing in QGC / MP. I’ve also verified the PPM input is working another way by plugging the PPM wire into an old APM 2.6 I had knocking around running some old version of Rover. Both work fine.

These tests lead me to think the issue might be in Rover-4.1.0-rc1 - Is anyone able to replicate this?

Thanks to everyone who’s worked on this release, amazing steps forward. Also can’t wait to try S-curve when that’s up for testing!

I have some very encouraging updates today.

First, to reiterate, BendyRuler does not appear problematic the way I described in my first post here. Again, it simply appears to be honoring the fence margin a little better than it may have previously.

As mentioned above, none of the things I tried, including minor modifications to the firmware code itself seemed to alleviate the brief instances of “No Fix” on GPS1 with a moving base/rover/GPS-for-yaw configuration. I even managed to get the SIK radios talking at 115200 baud, and that made no difference.

I used a slightly modified version of my ESP32 based serial bridge to get RTCM3 directly to UART2 of the moving base F9P, bypassing MAVLink injection entirely. I’ll post an update on that topic elsewhere, as ESP-Now range is quite impressive with external antennas!

With RTCM3 injection bypassing the flight controller entirely, I have extremely stable GPS operation - probably the best I’ve ever seen. GPS1 was a rock solid “RTK Fixed” for my entire testing session today under Rover-4.1.0-rc1.

It’s a bit frustrating that RTCM3 through the flight controller is so problematic, even with an H7 board, but I do understand the challenges there. Just like my testing of moving base RTCM3 to rover through the flight controller (GPS_DRV_OPTIONS=0), some rigorous testing of MAVLink RTCM3 to moving base appears to require some amount of compromise that may always be present. (EDIT: I was wrong…keep reading - MAVLink injection works great with F9P firmware 1.13+).

As of today, the way my fixed base station is configured, I now have a possibly unique arrangement that makes it extremely easy to switch between MAVLink RTCM3 injection and external/UART2 messaging, so that may prove helpful in future testing.

TLDR;
All that to say…I no longer see any roadblocks or concerns with respect to releasing Rover-4.1.0 stable, and I’m really looking forward to S-Curve navigation as a welcome update in a future release. Thanks much as always to the dev team for the attention to detail and receptiveness to feedback.

EDIT - the proof in the pudding - ~50 minutes of 100% RTK Fixed operation!
image

1 Like

I think the problem is that your F9P GPS has the older 1.12 uBlox firmware which has a number of significant bugs. You can tell that from these messages:

2021-09-08 01:53:02.940 u-blox 1 HW: 00190000 SW: EXT CORE 1.00 (61b2dd)
2021-09-08 01:53:02.940 u-blox 2 HW: 00190000 SW: EXT CORE 1.00 (61b2dd)

You need to use the 1.13 firmware (or later if available), which would have this
message:
u-blox 1 HW: 00190000 SW: EXT CORE 1.00 (f10c36)
the part in parentheses is the firmware versions.

1 Like

Allow me a moment to wipe the egg from my face while I reply…

It’s been quite a while since I last looked at the uBlox website, and I noticed the 1.13 update as I was writing my ESP-Now code, so I updated the modules this week while I had everything torn apart anyway…PRIOR to my last test of RTCM3 messaging.

So, I committed a cardinal sin in testing and changed more than one variable, then possibly mis-correlated the success in my last post with the RTCM3 injection method.

I will test MAVLINK injection once more this weekend with the later firmware.

I’m most frustrated that I KNOW better! I even recommend to others to upgrade the F9P firmware before doing anything with ArduPilot. I guess I was complacent for over a YEAR, thinking that I had the most recent on my own configuration.

Doh!

5 Likes

Yuri

Could you contact me to discuss an oppportunity?

jeff@autonomi-inc.com

Jeff

1 Like

Well, I’m sheepishly thrilled to report that you can disregard all after bonjour from me on this topic!

@tridge was 100% correct, as usual :slight_smile:. With MAVLink RTCM3 injection through Mission Planner and F9P firmware 1.13, I’m seeing rock solid GPS performance once again.

I suppose in this case, my own buffoonery was the mother of invention, and I really am encouraged by the results of the ESP-Now experiments, but that’s a topic for another thread.

@lutz, I sent you a direct message here on the forum.

EDIT: Here’s an example of just how GOOD BendyRuler can be. I’ve struggled to mow this area and usually resort to doing it manually. Today with good GPS performance and an understanding of BendyRuler’s limitations, I managed to make this happen:

4 Likes

@Yuri_Rage,

Thanks for the feedback!

1 Like