Mishap testing AC3.5.3

So, here’s a 700 class piston helicopter running custom firmware with several enhancements to it, forked from AC3.3.3. It hovers beautifully in Loiter and has flown the AC3.3.3 based firmware for over 175 hours on commercial flights without ever a single glitch. Looking at the IMU aliasing on the Y axis things don’t look too bad.

Same helicopter, no mechanical changes or changes to the FC mount, with AC3.5.3 in it. In Loiter it “toilet bowls” and won’t hold position properly. It has EKF issues with “changing lanes”, which causes radical upset to the attitude of the aircraft. If it happens close to the ground where there’s not enough altitude to recover it, it’s not good. I got damage on the blade tips when I was flying it just a couple hours ago. It decided to “change lanes”, snap-rolled 50 degrees in 120 milliseconds, and the main rotor hit the ground before I recovered it.

Looking at the aliasing on the IMU’s on the Y axis this is what we get, once again hovering in Loiter. There is no clipping indicated in the log but the IMU’s don’t agree.

Same helicopter, again no changes other than reverting to the old firmware and putting my spare blades on it (blade tips got damaged), and it’s fine again and back to its old self. So what’s causing this? One could presume an issue with the firmware and how it’s sampling the IMU’s. But I don’t know. Vibration is usually blamed, but the vibes are below 7 on this helicopter on all the axes, although the Y-axis does exhibit the highest vibration levels.

At any rate, I have only been able to test the newer firmware so far. I cannot fly it in my commercial helicopters because the EKF2 is not as reliable as the old EKF1 is. Maybe it’s because of the “shadowing” of the GPS receiver by the main rotor on a helicopter. I don’t know. But the EKF “lane changing” with a helicopter that can change attitude at 500+ deg/sec just plain does not work. I’ve had several incidents, and in-flight it’s recoverable. Close to the ground, when it happens, it’s not.

And I can’t come up with an explanation for it. Except for maybe “shadowing” of the GPS by the main rotor, and the fact that EKF2 is tied so tightly to the GPS.

I think I may have narrowed the possible cause of this to two things:

  1. the CDI ignition box for the engine might be causing some electrical interference. The compass wandered 70 degrees just before the EKF blowout.

  2. the proximity of the tailpipe on the muffler might be causing some low frequency vibration (sound waves) that’s not being logged but is affecting the primary IMU.

I can’t explain why it flies fine on the older firmware. But going thru some old logs shows that this unit has had some IMU aliasing in the past as well. And occasional compass issues, mostly when the engine is operating at 100% power output. Since the EKF1 in the older firmware doesn’t have “lane changing” it never actually “blew up” like it did with EKF2. But the telltale signs of a potential problem are still there.

Also, you should recalibrate compasses when jumping between those firmware versions.
I think levelling too…

Have you checked compass health?
I often have compass acting funny and mess up EKF (HERE GPS)
The compass reading will simply go off by ~100° and back to normal the other day. When this happens, EKF primary change occur several times in flight.

I did recalibrate the compass and the accelerometers. I had some issues calibrating the compass but I think the compass wandering by 70 degrees is what caused it.

The IMU aliasing seems to come and go. The Pixhawk is on the side of the frame and the engine is underneath and forward so the CDI box is about 10cm from the Pixhawk. And the exhaust tailpipe is about 12 cm away on the other side of the helicopter. It has a chainsaw engine in it. After thinking about it I don’t think the CDI box is causing compass intererence because the compass is a long ways away from the CDI. But I think vibration from the engine exhaust also affected the compass. Those chips are sensitive to vibration too.

So I’ve pretty much concluded that I need to rework the exhaust system and engine mount. And make a vibration isolating mount for the compass/GPS. It is very loud - it has a custom made tuned pipe on it with a baffle and stinger like an expansion chamber. Which seemed like a good idea at the time, but I should probably put the stock chainsaw muffler back on it, which is quieter and doesn’t have the sharp “bark” type exhaust note.

Ok, as a solution to this problem, I found the compass chip had been damaged. I ran the helicopter this morning in a tie-down at full power and in-flight collective pitch where I could observe it up close. The GPS/compass module was vibrating on its mount like a tuning fork. It had been running that way for awhile and the chip is no good. I replaced it and now it’s fine, although I have to make a better mount for it.

The GPS receiver seems to have survived fine.

I think the reason it worked with the older firmware is because maybe the newer firmware has better checks for the compass and it rejected errors from it sooner. I definitely got better offset values on the compass cal with a new chip in the GPS unit and the heading now holds dead-on after calibration.

I’m also going to change the exhaust system as I think that’s causing some issues with the IMU’s.

Could my compass got damaged as well? Any way to check?
It did experienced vibration but I don’t think it’s that much but it acted up randomly so I better check.

Pitt, look at the x, y, z outputs of the chip and what offsets are being used to calibrate. It’s a dead giveaway if it has wandered. Those mag chips can be damaged by coming in contact with a magnetic screwdriver, but they’re usually pretty tough. However, evidently they can’t take continuous vibration.

Update: after replacing the compass module, making a better rubber isolated mount for it and reverting to the stock muffler, the problems this heli had are totally solved. Test flew it this morning, early, in auto at 20m/s on a one hour flight (the last half hour in light rain) and it performed flawlessly.

So I learned something new related to vibration issues and the mag chip. Rather than having the GPS/compass on the tail boom I now have it mounted off the side frame where the mount can use the mass in the heli frame for damping. And it’s further vibe damped with a rubber mount that I stole out of a Honda weed whipper on the bracket and the module mounted to a 3/16" thick aluminum plate.

I was going to fly it on another 1/2 hour flight with a dummy 3kg payload @ 45mph to test for stability in fast turns with the 3.5.3 software (it had been flying 3.3.3-based software). But the rope broke on the recoil starter and it started raining harder, so I stopped flight testing it. I’m satisfied that it’s fixed.

1 Like