EKF blowout cause?

Had the heli with me up in the mountains yesterday, partly to try to get some scenic footage, but mostly because I could :).
However, shortly after switching it into loiter, it started strongly moving right, with the ground station complaining about ‘EKF velocity variance’ and I got EKF Primary errors in the log.
What causes these? Vibrations have gotten a little worse since I moved the pixhawk. Are they the cause of this?

Log:

Vibrations, I would say, is not the cause. You have some minor aliasing the IMU’s on the X axis, but not enough to cause a loss of attitude solution like that.

It was likely a GPS glitch. It was dropping in and out of 7-8 satellites and GPS speed didn’t agree with what the EKF State Estimator thought. So it “switched lanes” on the EKF - twice. I’ve had this happen before with both 3.4 and 3.5 in high speed flight, it kicks me out of my Auto flight, I switch to Acro (flybar), fly the heli in a big circle on FPV until the EKF decides everything is rosy again, switch back to Auto and continue with my flight like nothing happened.

I personally do NOT like EKF2. I’ve had more problems with it doing this than I ever had with 3.3 with regular old EKF. And it is one (although not the primary) of the reasons I’m using a FBL unit on my new FBL heli build. I don’t feel EKF2 can be trusted when it provides the sole attitude solution for the aircraft. Until ArduPilot comes up with a way to revert to gyros for attitude solution, I will not fly an expensive helicopter with it without using a redundant method of control (either flybar or FBL unit). Some will claim they’ve never had a problem. But I’ve seen it crop up enough times on helicopters that fly perfectly fine for 20-30 hours, and then suddenly at random the EKF2 goes on vacation and the helicopter goes nuts. It’s happened with multi’s too, resulting in “flyaways”, due to multi-path GPS signals (which is likely in mountainous terrain), mag anomoly, whatever.

That being said, your vibes are a bit high, especially on Y and Z. But I’ve learned that frequency of the vibes makes a difference too, on how much it causes aliasing of the IMU’s. I have far worse vibes than that on my piston heli’s, but they are low frequency vibes and don’t affect the IMU’s that much. My gut feeling is that your issue was caused by multi-path GPS signals in mountainous terrain, where the GPS time does not agree because the receiver gets a signal that takes longer to reach the receiver than what the calculations say it should - and that is the basis of calculating GPS speed. I’ve seen a Garmin hand-held unit claim I am going 600 mph before, standing in one spot.

1 Like

I suspected as much. My fixed wing plane struggled a bit with satellites too and so did my flying buddy whos Mavic initially refused to take off due to lack of satellites.

I have already started on designed a new mount for the Pixhawk, so that it goes back on the main frame of the heli. I suspect the arrangement it’s mounted on now actually amplifies the vibrations.

Thanks for the help as always.

Vibration is usually pointed out as the cause of EKF blowups. I don’t see that’s the case here. You had the same Velocity Variance that I’ve gotten on heli’s that have less vibration than typical multi-rotors. And that’s usually a GPS issue. The bad part, as I said, is that ArduPilot depends on it for the attitude solution, and has no backup built into the system. I’ve never seen yet where that EKF “lane changing” actually works. When it does it, it usually seems to select the EKF process that has the worst solution, resulting in a radical attitude change.

I view EKF2 as similar to the old adage with commercial pilots flying multi-engine aircraft; the chances of having an in-flight engine out compared to single engine is one times the number of engines.

So, on the bench, it’s perfect. 14 sats, 0.9 hdop (indoors). I get it outside and starts out great and then the GPS starts freaking out out.
I take it home, check all the wirings (seems fine) and then decide to change the GPS.
Next time out: Loiters happily for 10 mins, hdop 0.6, 12 sats, so I sent it out on an a dummy auto mission and it’s as good as new.
I then hook up the camera, send it out on the same mission and I can hear it making angry beeps, but nothing on the on GCS/Mission Planner? I land it in stabilize.
I strap a lighter battery on it and it does a weird failsafe. just as I try to take off. I get a EKF Velocity variance warning , but that’s clearly not it.
I take it home and the logs shows RC-IN failsafes, despite RSSI being fine?!?
I have now replaced the Pixhawk with a different unit.

So, replaced the Pixhawk with a different unit, loaded 3.5, but the problem still persists. (GPS drops out)

With the GPS being mounted on the tailboom and this being a belt-driven tail, could this be a static issue?
I’ll make a new cable in any case.

Jacob, it could be a static issue. If you suspect it, just install a ground wire from the tail boom to the battery ground. If the GPS puck is mounted right next to the tail boom, you could also try installing like a 20-25mm extension to get it up off the tail boom a bit. I do that on all my belt tail heli’s and have never had a problem with them.

I can relate this - way back on the 700 nitro I had a problem with vibration on the tail boom causing the compass to fail and GPS to be erratic with it mounted on the horizontal stabilizer. I moved the GPS puck in on top of the tail drive case where there’s less vibration from the engine and it solved it. I have since mounted the GPS puck there on all my heli’s with never a single problem from “shadowing” of the main rotor blades or head.

I’m building a new Synergy E5S/626 and the Pixhawk on that one will be mounted on the top the tail transmission. I don’t know what I’m going to do with the GPS puck on that one yet.

Yeah, I saw your mounting position on one of your recent videos and figured I might do the same. It’s currently raided about 10mm up from the boom.

After reviewing my log, I’m even more confused. I assumed it was the same problem, but the GPS was fine when it decided to drift severely to the side.
Getting a GPS mag error earlier as well.
I managed to destroy my fancy spinblades last weekend, when first testing the ‘new’ pixhawk. Someone forgot to make sure that the roll direction on the swash was correct :/. However, I put the ‘old’ Edge blades on and vibrations dropped by quite a lot!.

In any case, I’m at quite a loss of how to proceed from here, so if I could ask you to yet again have a look at my log file and see if anything stands out?

Log:


Params:

Jakob, your vibes look good. IMU’s look good. But I have a suspicion that you have the same thing I had going on with my piston heli the one time - being mounted out on that tail boom there’s more vibration there, and it’s shaking the living snot out of that compass and GPS unit. And it causes it to be erratic - I know that from experience with that happening. And because the GPS is erratic it sets off the EKF notices and the constant EKF lane switching.

If it was me, I’d try moving that GPS puck in on something more solid and try it again. It’s a real simple way to eliminate that possibility. I see nothing else wrong with it.

Darn! I hate when that happens :grinning:

Edge blades aren’t made anymore. But I bought a bunch of different 600 size blades at IRCHA just to try 'em out. One of those deals where you can’t leave a vendor tent without carrying some new toy :rofl:

My wife grew increasingly suspicious as my blade collection grew. So I told her I just want to try them out and if I don’t like 'em I can sell 'em on HeliFreak. I got stock EP Align’s, CF Align’s, Spin’s, RotorTech’s, and Rails.

I put my RC power meter on the 600 so I could measure watt-hours used, and hovered each set for 6 minutes and recorded the watt-hours. Then did it a second time to make sure the results were the same and not bogus.

Then I flew each set on my 1 mile long auto test flight that has climbing and descending turns, legs where the heli cruises at 30 mph, and it makes three laps of the course using Do Jump command. Measured the watt-hours used on each flight with the RC power meter.

The winners in both hovering and cruise flight was these. And actually the first time I had ever seen a 600-class, 4.68kg/10.32lb helicopter cruise on only 450 watts of power on the crosswind legs of the flight.

Don’t let anybody tell you blades don’t make any difference because the worst ones were 575 watts @ 30 mph cruise on the crosswind legs.

https://goo.gl/photos/yY8egq71t4A3nBQBA

Too bad that they don’t make those in my size (~470mm). I did however order a pair of VTX477’s. It’ll take a few weeks to make their way up here.
The Spinblades (asymmetricals) were definitely more efficient, but as mentioned, also caused way more vibration. I guess I need to get a balance stand too and not rely on the manufacturers to balance them.
The SB’s would hover at about 17A and the Edge blades are at around 19.5A

I’ll move the GPS. It’s certainly quick to try.

Thanks again for the help.

Chris, I am trying new blades from Germany from - www.derblattschmied.com. Those are called S-Schlag. It means they are totally flat underneath. My SpinBlades are not really flat underneath, still little round. I ordered a set of 3 blades, 760mm from’ Der Blattschmied’ (the blade smith ). Because I have that 3 blade head on my TR700E. I hope I can safe even more power with those. Not good for windy conditions. To much lift, I was reading somewhere. I am retired, I can wait for good calm day,s.

Hi Fred,
I have flown the Spinblades for two years (actually one set a cheap clone of them), and I can’t really say that I have had any problems with buoyancy in the wind with them. They are an asymmetric airfoil that is better described as semi-symmetric I think.

The Heli-Tech blades with flat bottom are quite popular with the scale builders for two reasons; they make them in CCW configuration, which most full-size heads rotate that direction. And the blade profile is closer to the real thing because real helicopters don’t fly inverted, and it allows slower head rpm so the main rotor sounds more like the real thing on the scale models.

The Rail blades are also not symmetrical. They have a quite interesting airfoil that I’ve never seen anywhere else with washout in the bottom part of it. It has a sort of “cusp” on the trailing edge underneath that washes out to symmetrical on the tips. And the tips have swept back leading edge and tapered trailing edge. They look the same as the RotorTech blades, but they are not the same in the way they fly. The fore and aft CG of the blade is different on the Rails and they are very torsionally rigid, which I believe prevents twisting and more precise “flapback” of the blade in the rotational cycle when cyclic pitch is applied. The result is that they run virtually quiet. I did a double take when I took off with them because there was just motor, geartrain and tail noise. The main rotor is quiet.

The worst ones were the stock Align blades. Very noisy tips, quite flexible blade, fatter airfoil. The Align ones are obviously made for 3D type flight and not efficiency. The fatter airfoil provides more “bite” for rapid maneuvers. But at the expense of power required to turn them. They consumed the most power both in hover and cruise, and have a distinct “slap” noise and more vibration in turns because they flex out of track. The Align blades are made to operate at extreme headspeed so the centrifugal force makes them track. At low headspeed (I was using 1,640 rpm) they are horrible. In the turns the Aligns went so far out of track that the helicopter sounded like a Huey going by. And it is obvious in the logs as well - with the Align blades on it the amps would climb to 27-28 in the turns, drawing 640-650 watts.

All helicopters probably can’t use the same blades due to disc loading, the speeds the heli is flown at, head geometry, etc… But despite all that, the stock Align blades DEFINITELY are not suitable for a UAV heli of any type. At least not if you want decent efficiency and a quiet-running helicopter.

Edit:
Forgot to mention another problem I had with the Align blades - a distinct tendency to bob the tail up and down in the pitch axis. None of the other blades did that. They would’ve required higher headspeed to settle them down. I left it at 1,640 rpm to try all of them because I didn’t want to skew the results, and just put up with the bobbing. But it was not suitable for UAV stability at that headspeed with those blades.

1 Like

Thank you Chris. I have printed out your observation of those Heli rotor blades. Very interesting!!
I don,t have any Align blades any more.

Unless you are into 3D I would say that’s a good thing :grinning:

So,Move the GPS puck onto the main frame and get it out in very swirly conditions, but it loiters reasonably well. I know I need to re-tune after the blade change, but it’s good enough.
I then send it on a simple auto-mission and it’s clearly confused.
I land and now I keep getting Compass Variance Errors. Maybe it’s struggling with the new compass/GPS location?
I try to re-calibrate and it wont. I then take a look at the Compass orientation and the external compass did not pick up the right orientation when I updated to 3.5!!. Somehow it allowed me to calibrate both compasses with one of them pointing the wrong way!. I correct the orientation, disable the internal for expediency, re-calibrate and it now flies happily. Argh!!!
It has also change the default speed and yaw rates in auto, but I ignored that and was just happy to see it fly again.

Do you think that was the problem all along? Or did you just change the orientation of the external when you moved it? Regardless, I don’t like putting them out there on that tail boom. There’s vibration there even on a smooth-running heli when the tail blades “bite” in a turn or whatever.

Not related, but I found a good use for old APM’s with an internal compass yesterday. I put one in a Trex 500L Dominator, no GPS/external compass, no MavLink - just using it for a FBL unit. Just two flight modes - Stabilize and Acro. It works perfect and handled hard 3D no problem at all. Cheapeast $30 FBL unit there is.

Now I’m trying to figure out how to get it to work with a Spektrum DX-6i with a Orange 615 receiver with CCPM output. Just thought I’d throw that out. If you got an old APM2.5 or 2.7/2.8 and a heli you want to use for fun flying. It was easier to get that APM tuned to match any of the modern FBL units than it is to tune a Pixhawk with the newer versions of Copter. Took all of about 30 minutes to tune it in and it “just works”. Those are some tough old boards, not near as finicky as a Pixhawk :grinning:

No, the original problem was definitely GPS glitches. Add the random radio-link failures, there was something going on, which is why I swapped out the Pixhawk unit. Similar, before the erratic behaviour was always linked to some kind of events, where as this time, There wouldn’t be any.
I also made sure that I mounted the GPS the same direction when I moved it.

As an aside, I saw on one of your videos that you were also using 10.000mAh Multistars on one of your helis; Could I ask you how much voltage sag you get one those? I’m getting almost 1.5v of drop, when pulling ~20A, which means I barely get 50% out of the battery before the voltage becomes too low for comfort.

It drops about dead on one volt with that MultiStar battery under load. The hovering amps is about 18-19. I have a power module on that one (don’t use the power modules on 6S and 12S). I don’t use the battery failsafe. But I have the low voltage warning set for 13.0V. If I fly it all the way to low voltage warning it recovers with the cells at around 3.7V. It has been a pretty good battery for that size heli. The only problem is that it’s pretty big and not a lot of options to mount it other than some sort of custom rack like I made in the landing gear.

They make those MultiStars in 5A too. I think the 5A one would fit in the normal battery tray under the canopy, then put another 5A one in a belly rack like that. I think that would be a better setup and allow to put the belly battery more under the heli. I had to set the 10A slightly forward to get the CG right.

Edit:
The battery I was thinking of is actually a 10C 4S 5200mAh MultiStar. The dimensions they give for it should fit in the battery tray on my 500 and allow to still get the canopy on. That is actually the design I wished I had gone with is two of those in parallel.

I’ve mounted my battery on a small sliding tray, so that I can adjust for the CG. It does mean mounting it ‘sideways’, but it works reasonably well. Pixhawk is now on the old battery mount. Wiring still needs tidying up after moving it around, but I wanted to make sure everything worked before making custom wirings.

Also designed a new canopy:

This is just a printed prototype, but I’ve printed a plug for a mold, which I should hopefully get made soon. (Making plugs is time consuming!!).

Also got my new VTX blades today. Should get to try them out tomorrow. If they fly as good as they look, they’re going to be amazing :smiley:

That is an excellent setup to adjust the CG. Really cool looking canopy too :grinning: I LIKE that. The stock canopy on most of these heli’s doesn’t allow much room under it for stuff.

I think you’ll notice that when you yaw the heli in Loiter it won’t tend to move around in a circle like somebody is swinging it by the tail with the GPS mounted in closer to the mainshaft. At least not as much as they tend to do when it’s mounted way out on the tail boom.

You don’t get any twitching on your rudder servo with the MavLink radio mounted there? I’ve gotten servos to twitch before just running the MavLink wiring alongside servo wires when the servo’s don’t have any ferrites on the servo cables.