Running TradHeli v3.6.10 (1c04a91e) and ran a proof-of-concept mission beyond RC range (2 miles configured). Made sure that RC failsafe action in OpenTX is to send zero packets. Made sure that both FS_GCS_ENABLE and FS_THR_VALUE were set to “2” aka “Continue in Auto.” Sent copter on its way and observed from Yaapu in the RC as well as Mission Planner. When Mission Planner lost connection, the copter continued. As the RC connection got spotty, and telemetry updates were not coming back as often, I’d get the callouts about RC Signal being Critical, etc. Eventually the copter RC failsafed, but here’s the weird thing. I saw one message in Yaapu saying that EKF had failed over and then the mode changed to LAND and then I didn’t get any data for a bit again. I tried forcing RTL from the RC but that didn’t take and eventually we stopped getting the spotty telemetry back. A drive down the trail followed by another mile hike got us close enough to hear the heli beeping as I was changing modes, but it was in the rough so I couldn’t find it exactly… I went for broke and armed the heli and took off and put it in to loiter and flew it over to me. I then clicked RTL so it would go back to exactly where it took off from, but before it started its descent I commanded Loiter again and it just hovered, where I recovered the broken parts, and landed the heli to walk it home. Surprised it was flying after the landing it had, which was apparently a 4.3G landing. The 106mm tail blades were probably 60mm or so, and the main blades had damage but were mostly intact. The landing gear was not really a thing anymore, and the velcro holding the batteries in to the metal slider had snapped, so the batteries were just “in the frame, chillin.”
Here’s my question: Why did this happen? This appears to be a Motor Interlock trigger at the same time as RC failsafe. I had a chat with @Leonardthall and he asked me to post this here and tag @bnsgeyer and @rmackay9 for a request at an additional log review to verify this was in fact a motor interlock being disabled at RC failsafe, or something else. Was it bad timing/luck? I know the log says the PWM values went to 874, but that isn’t a value my radio can send and I did verify before posting that my FS action to send no frames.
Here is the log, curious to hear what smarter people than me can come back with.
@JoshW I don’t know much about the failsafe settings and behaviors for sbus receivers. I guess the big thing to check before doing any flight where you anticipate an RC failsafe would be to check that your receiver continues to pass a good value on the motor interlock RC in channel to ensure motor interlock remains enabled. If you could do a bench test where you disconnect your motor power and arm and enable motor interlock. Then turn off your transmitter and see what the pwm value is on the rc in channel for motor interlock.
It needs to stay at the same value that but was before turning off the transmitter.
Edit:
I looked through both 3.6 and 4.0. It appears that in 3.6 and earlier that heli sets motor interlock based on RC channel 8. In 4.0, it appears that the motor interlock would not be affected by rc receiver sending no signals. I will have to test that.
Alright, thanks Bill. I am marking this as solved then, as it sounds like you’ve confirmed it was a vulnerability in pre-4.0. With 4.0 being released, I’m going to upgrade this Heli (same one we’ve talked about before, FWIW, not the uavcan one) to 4.0 now and start tinkering with the Governor too. Also with the RPM sensor I have installed, I can also tie it to the dynamic notch. Sounds like 4.0 all around will help me with this POC for bvlos as well as all the other things. Very much appreciate you taking a look!
I’m sorry that you had to find that the hard way. I will make an effort to update the failsafe wiki to note the vulnerability in 3.6. There is still a vulnerability in 4.0 is you use the RSC passthrough mode. The setpoint, throttle curve and governor should not be affected by an RC failsafe.
Yes, 4.0 has some great new features. Happy New Year!
Sounds good. If I can help, please let me know. Also, I saw in the params when setting the failsafe actions to 2 that the description called out those actions being deprecated in 4.0 What will the new method for continuing on RC and GCS failsafe be?
Agree! Although, being the one that found and proposed the fix for CUSTOM_ORIENT when it asked for degrees but was processing as Radians, I wish the release notes would have called this out, that if you used custom orientation before that there was no automatic update, and that you needed to reset that yourself. Hopefully it doesn’t get anyone though, as I believe most custom settings will cause the vehicle to not arm.
@Leonardthall I believe this only pertains to heli’s because In Copter 3.6 and earlier we use the output of RC channel 8 to set the motor interlock. This was removed in 4.0
I believe this only applies to like really old Futaba radios, or similar. All modern radios using SBus set a failsafe bit with “no pulses”. The receiver will set that bit, the flight controller “knows” there is a failsafe event because of the bit that is set to 0. And the flight controller responds with whatever it is programmed to do.
Was the receiver actually configured for no pulses? Because that is not the default, and it can’t be set from the transmitter.
We have since gone to the R9-series on 900Mhz that have 5+ miles range, but we used to lose signal sometimes with the X-series receivers and never once had an in-flight shutdown with 3.6.
Hmmm… The outputs should not have gone to SBus zero.
With any D-series, X-series or R-series receivers on SBus the failsafe bit should be set and the channel outputs “freeze”. Doesn’t matter if you chose No Pulses, or Hold in the transmitter as long as no pulses failsafe is set in the receiver.
I looked thru our testing logs and I found one where this is demonstrated. The test pilot ran the AutoThrottle up to 1462 pwm with the heli running at flight idle on the ground. Then she force shuts the RC radio off. The system disarms right away and shuts the engine down because it’s landed. But the AutoThrottle input stays at 1462pwm, regardless. You can unplug the receiver and it does the same thing.
The slope in the line is deceiving, as that is the way the software graphs it to the next instance where the RC is enabled
Then she turns the radio back on and establishes a link again, restarts the engine, engages the clutch and gets the rotor turning at low idle, but doesn’t actually engage the AutoThrottle. This is a test to see if the throttle position “holds” and triggers the proper failsafe message in various throttle control stages. She shuts the RC radio off, it disarms and now we get the “Check Throttle Position” message because the system is in failsafe and the throttle control is still at 1264 with no RC connection.
This is with a R9 radio system, but is the same for the D16 and X-series receivers. You cannot configure it to do what it did in your log.
The SBus failsafe bit is used in the PX4 HAL.
I don’t know how ChibiOS handles it, or even it handles it correctly. We do not use ChibiOS, have no plans to use it, haven’t tested it. But what you had happen is an impossibility with the system we are running, which is based on the same code (except for the RTOS).
I just checked the log from Josh, and for the firmware version he was using the RC input was being handled by NuttX, not ChibiOS. This release was from before we switched the IOMCU firmware across to ChibiOS. The RC input on this board (a CubeBlack) is handled by the IOMCU, not the FMU.
Edit: this post for user information, Jan 10. I tested the RC failsafe on ChibiOS and it works fine. Bench testing for RC failsafe by simply turning off the radio is not a good way to test it.
Put your radio in range check mode with the system armed, then walk away to let the signal drop prior to failsafe results in expected behavior from the control. Turning the RC off with the system disarmed on the bench results in different behavior that you don’t get in the field when it’s armed.
Both. First identified on my firmware. Then with 3.6.12. Loaded the ArduPilot ChibiOS bootloader - that didn’t make any difference. Built the PX4 bootloader and reloaded that. Tried a reversion back to a build (on my firmware) before the scheduler was modified. That didn’t make any difference.
At that point didn’t have time to further debug it. It dd not pass the required pre-flight checks, so we went back to NuttX which obviously works fine.
It appears the IOMCU mixer has been hard-coded to put the aircraft into a hard left spin if the FMU goes down in RC failsafe (or possibly some other configuration), which is not acceptable. It’s not acceptable to have the FMU go down in flight in the first place.
Leonard, I forgot to mention I did try it with a fmuv5 too, and on that controller the problem did not exist. For now we have to stick with what we know works or we’re grounded.
On one instance the controller refused to boot at all with the ArduPilot ChibiOS bootloader. That’s why I put the PX4 bootloader back in it. Once I got it running again on ChibiOS the servos go to an extreme position on boot (randomly), which doesn’t hurt anything on the helicopter in the vid, but it will break linkages on the other ones.
I plan on investigating the problem further when I get time. I want to write a pseudo “mixer” that allows the pilot to retain control of the helicopter in autorotation if the FMU quits. Rather than in the IOMCU firmware that mixer will likely be in the RC radio, switched in with an emergency mode switch. But the IOMCU has to obey the basic control inputs on four channels to make it work.
If we can get a reproducible issue that will be good because NuttX is an extremely compromised system and will not support a lot of the higher processing capability that we are now able to use on ChibiOS. Also because ChibiOS is much more streamlined it is much easier to work with and debug. So any reproducible issue will be quickly addressed. The problem is your example is the only time that we have seen this issue so I suspect it isn’t easily reproducible.
It may be practical to use NuttX on 3.6 based firmware but I it is no longer being supported so I would not trust in on 4.0. I have not used NuttX for 12 months and will never fly with it again.
NuttX is supported just fine, including on fmuv5, and is worked on by an active team supporting PX4-based hardware, which is what most of the autopilot hardware is based on.
I think Greg updated to NuttX 8.2 a couple months ago. Nothing is ever truly unsupported in open source. It forks. It evolves.
