Sorry to hear about your hard landing. Sounds recoverable.
Indeed, the EKF blew up and “switched lanes” a couple times. It says it had a GPS glitch but you had good sats yet. The second time it blew up it set off the EKF failsafe but that time it had a huge 70 degree discrepancy between desired and actual roll. The IMU aliasing does not look all that bad at that point. It appears to be more of a GPS issue causing the EKF to go on vacation, and I don’t know why (although I do not particularly care for the latest implementation of EKF2/3 and is why people are flying downstream FBL units with 3.4 and later).
The GPS glitch could be caused by multi-path signals being picked up by the receiver, although I can’t say for sure that’s the case here. It still should’ve been flyable in Stabilize flight mode, although you would have to fight it back onto the ground.
Chris, I looked at his log. I don’t know much about the EKF errors but it appears that an error occurred earlier in the log. The error code says failed to initialize. During the flights like you show in your plots, two additional EKF errors occur one showing that it was resolved and the other was a failed to initialize error. During both of these events the desired attitude tracked with the actual attitude of the aircraft. I looked at the NTUN message which contains navigational information and there appears to be some discontinuity in the desired velocity signal. I think that is what’s causing the aircraft to put the aggressive maneuvers in I still believe that this is a EKF issue since the velocity of the aircraft is estimated by the EKF. It could be instigated because of the GPS error which we have no information. The last large roll that goes to 70-ish degrees at around an index of 37,200 was when the aircraft hit the ground (baro alt goes to zero).
That’s all I can figure out for now. Maybe someone else that has better knowledge of the EKF can tell us more.
I saw that too and that’s what led me to believe it might be multi-path GPS signals causing it. Velocity errors with GPS measurements are most times concurent with multi-pathing of signals and the time stamp being off.
GPS glitches can and do happen, they are not rare by any stretch, and can cause any aircraft or vehicle using GPS navigation to do strange things. The usual recommendation would be, if flying in a GPS mode like Loiter or Auto, to get into Stabilize or another mode that does not use GPS. I have had this happen before where I get the EKF Velocity Variance (the log usually notes that), and it kicks me out of an Auto flight. But I have the default EKF FS = Land disabled because I can’t afford to have my helicopers landing in trees or whatever every time it happens. It kicks me out and into Alt Hold, I usually express my disgust with the situation, switch to Acro and fly the helicopter in a big circle until the EKF decides everything is fine again, then flip back to Auto and continue my flight.
I see no reason this helicopter would not have been flyable in Stabilize flight mode in this case. I have experienced where the EKF blows and I lose the attitude solution and the system thinks the aircraft is banked at 45 degrees or something when it’s really level. And you really have to fight it when it does that and flying it visually and forcing it to do what you want is really the only way to fly it with the EKF gone.
As I mentioned, this is the reason Marcus and several others won’t fly 3.4 and newer without a downstream FBL unit. In Acro flight mode the downstream FBL unit gives them redundant control method when the EKF “switches lanes” and causes an extreme change in attitude or loss of control. And I really don’t know the answer to this.
Unfortunately, Geoff, I would have to conclude after reviewing your logs further that vibration is likely affecting the performance of the gyros and causing the EKF issues. I’m seeing almost consistently around 40 on the X and Y. Generally (I think) 15 is considered the max for acceptable performance of the gyros, and up to 30 on the Z-axis.
You could play back the tlog from the flight with the vibration monitor on the screen and see if it shows excessive vibration and EKF problems related to it. I’m thinking it almost has to. That is pretty severe vibration for the Pixhawk.
Edit:
I want to add that vibration can also cause issues with the GPS and compass chip.
Hi All
Well I got my 550E Heli repaired, with the shorter front legs, which gives me more tail clearance, and the front tail gears changed, I also altered the Pixhawk mounting, from foam pads to the four silicon balls.
So, I did a little test flight in the windy garden. I would appreatate if you would check the bin file, to see if I have got rid of the faults that you found on my crash bin file
Geoff, your vibes are now at very acceptable levels and the log looks clean for EKF errors. With the lower vibe levels getting to the Pixhawk I expect it will fly much better and the IMU’s will be able to better measure aircraft attitude without all the vibe “noise”.
Thanks Chris, I feel much better for that. When the wind drops, I’ll try it on the field, a lot more room there. Keep up the good work, helping the Trad Heli flyers. That includes you Bill
I got to the Field to test my 500 Heli. I had a GoPro camara mounted on the front Skip frame The first Bin file was on a Auto flight when it developed a ROLL shake, I then switched to RTL. The Heli landed safely. Attached the video from the camera.
The second bin file was a test in Loiter, without the camera to check for shaking. It now looks OK.
Could this be caused by the Pixhawk floating on the ball mounting?
No, it is not the Pixhawk moving on the ball mounts. It definitely went into a roll oscillation due to, I think, the weight of the camera causing an off-center CG situation. Your tuning looks really good too. I think what I would do is mount the camera to that helicopter and don’t do a full re-tune, but drop the D gain in roll to zero. Hover it in Stabilize with the camera mounted and aggressively shake it in the roll axis with the sticks and see if you can set it off again. Or maybe brush a skid hovering sideways and see if you can set it off.
If it still breaks into an oscillation drop the rate P gain on roll from .05 to .04 and try it again.
I’m almost certain it is the camera weight off to one side, and possibly that along with some wind resistance in flight, causing it.
I think you used the new tuning method on that helicopter so you must’ve found the roll oscillation point at 0.1 rate P during tuning. So it SHOULD handle a wide range of wind and/or CG and still be stable and not over-tuned. So that being the case, check your servos for one that might have a bad pot too and went into the “jitters”.
Just last weekend I was flying my 626 and lost the tail servo in Auto. Brand new BLS251 servo. I had a wild battle with it for 1 minute and 14 seconds, the log showing the helicopter spinning in yaw at 1.5 times/sec. The autopilot instantly lost control of the helicopter and I got it back in Stabilize. I finally got it over a level grassy area where I could land it. I cut the power, which stopped the spin, and autorotated it. When I walked up to it the tail servo was still quivering and refusing to respond to the stick. I grabbed the servo horn and moved it and suddenly the servo “grabbed” and started working again. Futaba wanted more to send the servo in to get it fixed than I could buy a new servo for. So I just replaced it.
I don’t think your issue was a servo because I see in the log where the RC.OUT on channels 1 & 2 appear to be actually driving the oscillation. But the Pixhawk would do that even if servo 1 or 2 momentarily “locked up” or went into the “jitters”, trying to compensate for the rapid oscillation. So I mention the servo issue in the event that detuning the D and P gain further still doesn’t seem to fix it. Because weirdly, the desired attitude in roll looks pretty smooth.
The other thing you could try, since you have good settings with no payload, is to try another tune with the camera on it. And see if you end up with different settings.
Watching your video kind of indicated it went into that oscillation due to partly wind loading. It appeared to sort of come out of it depending on which way the heli was pointed at some times. Does that agree with what you observed from the ground?
And since it seemed to come out of it in hover, and only happened in forward flight, just wondering if you’ve tracked the blades on your helicopter? That is another thing that could cause that.
Chris, I managed to do a little test this morning. With the Camera mounted as before and the D set to zero. I could not get it to oscillation. So I will have to wait for the good weather to do a full test flight.
I tracked the blades as I was setting it up, after the crash the other day
That flight looks much smoother. Gave a chance to look at your rate peaks too when you were rolling it, which don’t look too bad. So I think the VFF is about right.
Hopefully, just dropping the D gain will fix it. The attitude tracking in roll looked right on when you were rolling it back and forth.
I notice you are running a -v3 firmware. This must be a Pixhawk 2.1?
The Pixhawk’s I use are V2.4.8. The latest Pixhawk 2.1 is still a bit costly, and awkward to mount by it’s size. As I only do pleasure flying, the older Pixhawk does me fine.
Hi Chris, I did a longer test flight using LOITER. It looked fairly stable, It was a bit swirly windy, in the garden. Looking at the IMU’s I noticed that the ‘Y’ is not lining up as much as the ‘X’. Vibrations look good to me. How can I improve this? The wind here in the south, UK, has been bad for model flying this year, especialy setting models up.
I see that as well. But it looks to me like at lower vibrations it deviates no more than about .8. And as y-axis vibes increase thru flying it, or whatever, they then line up quite nicely. So being there’s no errors and the tracking actually looks pretty good, my opinion is that it’s a non-issue as that is very, very low deviation or aliasing.
It could be the fact that the Pixhawk is mounted on its edge on the side of the frame. When I started looking at IMU aliasing with a recent problem I had, I notice somewhat the same thing with my side-mounted Pixhawks.
Hi Chris, well I got a auto test flight in to-day with my 550E heli. I had a camera ridged mount to the leg strut. I appearing to be getting a slight PITCH shake, during this test Would I alter the D value down a bit to clear this? The ROLL shake seeme to be cured. Please find attaced Video and bin file, to give you both sides of the picture.
Geoff, that is actually very good video for a solid mounted camera on a landing gear strut. I think what I see in the video is in-flight vibration from the challenges of such a mount on the landing gear. I’ve mounted cameras there before and they typically take very good still imagery but normally a little rolling shutter or “jello” in the video from airflow over the camera shaking it some. Plus the landing gear is a challenging place to put a solid-mount, non-vibration isolated camera. So I think that’s actually pretty good.
From the log I don’t really see any problems with P and D on the pitch axis. But I think your I-gain is low. Consider this plot of pitch attitude, desired vs actual:
You’ll see I have your current I-gain settings highlighted there. Some considerations on I-gain tuning;
your flight speed is less than 5m/s. So I think the I-gain is being leaked off and is not fully active. You have to be going at least 5m/s sec for more than 2 seconds to turn off the leaky integrator.
To help alleviate this problem in hover and at low flight speeds < 5m/s you can set the ATC_RAT_xxx_ILMI parameters to a non-zero value. I would not recommend using more than 0.1. And probably start by trying maybe 0.02. The downside to carrying too much I-gain leak minimum is that it can cause the heli to want to tip on the ground or during takeoff if the buildup of the I-gain is too high. Helicopters normally require less (or even zero) I-gain in hover or slow speed flight. But as the speed picks up they need more I-gain. So the attitude controller is designed to provide the variable I-gain by “leaking” it on the ground, in hover, and slow speed. And when the heli gets moving it “kicks in” and starts working.
I think your I-gain settings (ATC_RAT_xxx_I) are pretty low. You are at the suggested starting values by setting them equal to VFF. But most helicopters like more. I don’t think any of my FBL ones are below 0.2. The flybar ones I typically put in 0.12 - 0.15 because it makes them track attitude nicer in fast flight.
You’d almost to have to fly the heli faster than 5m/s to see how it tracks once the I-gain “kicks in”. But I think probably .20 at least will be required.
This is an example of what it should look like. This is my Synergy 626 flying at target speed of 30 m/s (~110 km/h) making left and right turns around a waypoint test course in a figure 8 pattern. It is decelerating for turns, then accelerating again down the “straights”. It was hitting 97-100 km/h before decelerating for the turns, as I had the horizontal accel limit set to 3.5 m/sec^2 and max frame angle limited to 25 degrees at the time of this log snapshot. You can see how accurately the pitch and roll attitude tracks desired vs actual. This is actually a combination of having the proper rate (VFF, which I think is ok on your heli), proper P and D tuning (which I think is ok on your heli), and carrying enough I-gain to provide the longer-term attitude tracking accuracy in high-speed flight.
You will see where I have it highlighted what my I-gain settings are to make it fly like this, and I have the leak min set to 0.02 for both axes:
