[SOLVED] High EKF variance failsafe - cannot reliably use manual or auto modes

ArduRover ArduBoat
1

Hello everyone,

I am currently working on converting a commercial RC boat to autonomous using ArduRover (boat) on a Pixhawk Mini. Here is the vehicle: https://www.emilyrobot.com.au/. Also, I did not actually integrate the Pixhawk with the hardware. This was done by others, but I am setting up the firmware and doing the testing. Its not my boat, so I can’t cut into the hull. The Pixhawk, GPS, RC receiver, and Sik radios are all in a pelican case together. I think it is not ideal to have the GPS in the case down inside the boat’s interior.

In the lab, the only issue is that the compass calibration is reported as medium. Displayed right in the middle of the yellow bar within QGroundControl. I tried multiple times to calibrate, but the boat is a bit unwieldy to turn on all those axes smoothly.

I have conducted three short field trials. Note that I am operating in the Laguna Madre in the very windy Corpus Christi, Texas. There is substantial current and wind acting on the vessel. In the first test, I only tested manual control. It worked, but with two issues. One, the range was limited to ~280 feet. I even put the GCS Sik dongle on a tripod. More critically, it was very often going into hold mode due to “high EKF variance”. Then it would be “initializing” for ~30 seconds before I could regain control for a little while.

In the second test, I tried waypoint following and RTL. Both worked, but with substantial weaving. I am not worried about this since I know I need to tune it. It takes a lot of thrust to make a sharp turn. However, I was still experiencing the “high EKF” and initializing when using Manual control.

Today was the worst test of all. I went out with the intention of tuning the control, but was unable to maintain any modes for more than a second. As soon as I applied any throttle in manual mode, high EKF variance and initializing occurred. Whenever I tried auto, guided, or RTL… same thing. It was windy all three trials, but today was by far the windiest. The vessel pitches and rolls considerably. Perhaps this confuses the EKF, but at the same time I expect people have been using ArduBoat in rougher conditions.

So, basically, does anyone have any insights regarding this high EKF variance issue? I have seen several builds where the Pixhawk and GPS are down in a Pelican case, so I think it should be usable even if its not perfect. Basically, I am not sure if I should focus on better placement of the components or if I am missing something obvious that I need to address. I can worry about radio range and tuning later, but I can’t do much of anything if I can’t maintain control. Let me know what additional information I need to provide.

Once I have this boat working smoothly, I will integrate my sensors and autonomous mission planning software. I have code for onboard seagrass classification, water-currents aware path planning, mono-vision collision avoidance, etc. I’ll be happy to share my progress.

Thanks in advance!

2

Cool boat,

EKF variance means that all the sensors are not agreeing, this can be due to many things. A log will tell us what is going on.

3

Thanks much for the prompt response! It is a very capable boat, so I think there is a lot of potential.

I have not analyzed any logs yet, but here are the logs from today. I might be blind because its not obvious to me how to pull these up for analysis in QGroundControl. So I haven’t inspected them myself yet; I just checked the file dates. I can grab logs from previous missions as well, but I don’t want to bombard you with them. The last couple are probably just from turning on the vehicle at home.

ardurover_logs_ekrell_20200629.zip (283.2 KB)

EDIT: I may have made some progress. Like I said, the Pixhawk and GPS/Compass are housed in a Pelican case. I considered that perhaps some interference was at play, with so many components crammed together. Your comment, “sensors not agreeing”, gave me an idea. The Pixhawk and Compass are crammed in the case, but not exactly secured. I just stuffed a piece of foam to keep them in place. I believe there is a strong possibility that they shift a bit when I open/close the case, and possibly while underway. I assume this would count for their not agreeing since their relative positions/headings change.

I just now got the Pixkawk and compass out of the case, bound them tightly in a support, and re-did the calibration. For the first time, the compass calibration reports “good”. I am moving the pair quite vigorously and the EKF errors are < 0.0 when still. The compass_variance maxes at ~0.2 when really fast. I plan to test out the new setup tomorrow morning.

1 Like
4

Today’s mission went well! I did some manual control and a quick auto mission. Worked perfectly. EKF variance stayed low. So securing the compass and pixhawk tightly in the Pelican case did the trick!

Huge thanks @iampete. Just a quick comment from you was enough to get on track. I was wrongly interpreting “variance” as meaning that they compasses were reading something different that what they expected to. As in, the wind/current making it vary w.r.t. how it would behave in calmer conditions.

2 Likes
5

Hi, were you able to completely fix the issue just with securing the items inside the case even with choppy and windy conditions?

6

@drinnird

My tests so far have been windy with a current present. If I set the boat to hold, it drifts but it hasn’t been very choppy. My (naive) understanding is that it should be okay. The problem was that the two sensors disagreed; as long as they are moving together, I expect it to be okay as long as the conditions aren’t too extreme. However, I have no idea what the threshold for “too extreme” is.

Two weeks ago I got the PID tuning going well. Tomorrow I have another field trial planned to test out smart-RTL and loiter. I haven’t been careful with noting weather conditions. I will try to do so, and will report back if I get some rougher conditions.

I am also blogging my experiences (https://ekrell.github.io/), but keep in mind that I am bound to make lots of naive statements. This blog is really intended for myself to keep notes and share my progress with others in my lab.