Unexpected Heli RSC to Idle while in Auto Mission, FS_THR was correct?

[carlosanlley (Carlos Sanlley)]

This happen in a 700 electric helicopter built as a mapping platform powered by a Pixhawk 2.1. It has around 40 flights after completing the tune phase in different scenarios, altitudes and with different payloads.

A couple of days ago I took it for a flight and half way into the mission the heli lost power during level flight, taking it to a uncontrolled descent that ended in a crash.

Reviewing the mission planner I notice that the “No RC Receiver” message was displayed a couple of times and initially had no relation to the incident. I normally set up the FS_THR to 2 “Failsafe Enabled Continue with Mission in Auto Mode” because my typically mapping project will need for the heli to go beyond line of sight, and it will be usual for me to get the “No RC Receiver” in the screen while flying with no repercussion whatsoever.

In the post crash inspection, all wiring of the servos, esc, bec, batts and motor looked ok with everything connected and no broken wires. Looking into the log file did notice that a couple of seconds after the 1st Radio FS error, the Ch 8 aka Heli RSC had a pike downwards (graph #1), but after the 2nd Radio FS error, the Ch 8 went to 0% (1100 pwm) for about 2 seconds (graph #2), coincidentally at the time of the power loss (graph #3).

Graph #1, RCIn Ch8 vs RCOut Ch8

Graph #2, RCIn Ch8 vs RCOut Ch8

Graph #3, RCOut Ch8 vs Baro Alt

Any idea on what might have caused Ch 8 going to 0% even with the FS_THR set it to what it seams to be the correct parameter? It seams that the radio signal loss had some effect in the Ch 8 behavior.

Digging some more into the logs, ESC log showed an error msg that said “Loss of Radio Signal”, that might confirm the actual Ch 8 behavior on the pixhawk…

Anyways, I will review all the information with more details to see if i’m missing something, and bellow is the link to the google drive where I have uploaded the logs, telemetry logs and parameters files.

https://drive.google.com/open?id=1evyyJIFPYXc4e94d3LHbYo8OcVS7B-CM

BTW, the same mission was flown a week before, with the same “No RC Receiver” messages during auto operation but without any anomalies in the Ch 8 output (log #65 in the google drive is frome this flight and log #66 is from the flight of the crash).

[ChrisOlson]

Do you have your RC receiver set to hold last value on Channel 8 during loss of signal? This must be set in the RC. Helicopter always obeys the throttle signal from the receiver no matter what. So if there is late frames from the radio and the receiver is not set to to maintain the throttle signal it will kill it.

The failsafe monitors the collective and the collective going below a certain value (usually set to no frames in the RC) will trigger it. It will continue just fine as long as the receiver also doesn’t go to low output on Channel 8 during loss of signal.

I can’t look at your log because it says I need permission to see it.

[carlosanlley (Carlos Sanlley)]

Hi Chris, I have updated the permission of the shared link so anyone can access the files without needing to sign-in. Please give it a try again and let me know if have any troubles accessing it.

[carlosanlley (Carlos Sanlley)]

@ChrisOlson

Do you have your RC receiver set to hold last value on Channel 8 during loss of signal?

Yes, actually if you take a look to the flight log #65, you can notice that the radio signal is lost a couple of times without any further alteration of the flying state, Auto flight is maintain, Heli RSC or Ch8 output remain the same during the period of no RC signal and after.

Between these two flights no changes were made to the Heli, RC Radio or parameters.

@ChrisOlson

So if there is late frames from the radio and the receiver is not set to to maintain the throttle signal it will kill it.

I will double check that now on the bench to see the behavior of the Ch8.

[ChrisOlson]

It definitely shut down the throttle to the ESC:

And then the altitude decayed. Even though the throttle came back a couple seconds later the collective was maxed out:

When the throttle hold came on by itself the collective momentarily went to autorotation pitch. It looks like you’re running maybe 2.5 degrees negative or so. Then the throttle hold went off again (motor interlock in ArduPilot) and the pitch maxed out because the system was trying to maintain altitude. But there was no power even though the the throttle signal came back

So at this point we’re not in autorotation anymore - the helicopter’s main rotor has been stalled by the autopilot and the heli is in drop like rock flight mode.

For some reason your ESC failed to re-initialize after the loss of throttle signal so the power never came back.

So the question is, is this a Taranis radio? There is/was a problem in the way ArduPilot handles SBus combined with the way the Taranis (or OpenTX) rebinds on loss of signal, causing the throttle output from the receiver to drop. Seems to me I saw some developer discussion on this on GitHub and I don’t recall if this was patched in 3.6 or not. So the first question is if this is a Taranis or Horus with OpenTX, then I’ll dig a little more to make sure this has been addressed in the later code.

For all practical purposes here, the throttle signal should not have gone to zero in the first place if the receiver is set to hold last value on channel 8. That is a problem in the RC on how it handles the rebind when the RC comes back. Since the throttle did come back, but power didn’t come back, there was a problem with your ESC not going back to full power.

So looking at your settings, it appears the SERVO8 range is set to the defaults of 1100 to 1900.

Most ESC’s want 1000 to 2000. So is your ESC calibrated for this 1100 to 1900 throttle range? Does it have an autorotation window that was triggered by the change in the throttle value to 1100 pwm, that cause it to not return to full power when the throttle signal came back?

Best as I can tell, there should’ve been only a momentary loss of power when the throttle dropped (related to the ArduPilot/SBus/possible Taranis issue), but power should’ve returned regardless, as the throttle signal to the ESC was at normal power all the way to the crash. And it appears your ESC failed to return power to normal.

[ChrisOlson]

Upon closer examination of the sequence of events, it appears the problem is in your RC, which caused the throttle signal to go to 1100 pwm. At 608.465 seconds the radio failsafe was cancelled after ~4 seconds of no signal from the RC. The collective value came back to 1547 pwm, but throttle signal (RC8 IN from the receiver) stayed at zero.

So at this point the controller did was it was told to do by the RC. There is no radio failsafe, the RC8 IN is at zero, hit throttle hold (disable the motor interlock). A few fractions of a second later the throttle signal came back on from the RC. So the controller once again did was it told to do - re-enabled the throttle signal to the ESC. Now the ESC made things worse by not re-initializing (possibly due to range calibration problem, autorotation window set, etc.).

I also notice that normal range for RC8 IN should be the full range of your RC cal on channel 8. Your cal is 1099 to 1901

But the signal only goes to 1500:

So this brings up a question about having inadvertent mixes in the radio that may be causing a problem with why the throttle signal did not come back at the same time as the collective signal when the radio failsafe was cancelled upon re-bind.

[bnsgeyer (Bill Geyer)]

@ChrisOlson I am concerned that there is more going on here that we don’t understand. From what I can tell, his controller is not set up to hold last value on all other channels when transmitter link is lost. However looking at the logs, you will see that RC8in goes to zero during the time of lost link however the interlock never switches. It only happens just after the link comes back because RC8in is still zero for a fraction of a second which causes the momentary interlock toggle in one instance halfway thru the flight. I was trying to look thru the code to see if during transmitter failsafe events that the software is doing something different with the RC inputs. No luck yet.

[ChrisOlson]

@bnsgeyer

It only happens just after the link comes back because RC8in is still zero for a fraction of a second which causes the momentary interlock toggle in one instance halfway thru the flight.

Yep, that’s why asked if this is a FrSky radio with OpenTX. It sometimes takes 4-5 frames during rebind from lost signal to get all the channels back.

If you recall, back in Copter 3.3 Rob had a patch to prevent this, as the SBus would go to 874 with zero output

Improve handling of SBUS Failsafe

We had Rob’s patch in ArduHeli 3.3.3. When 3.4 came about all this code changed, and I had thought we put Rob’s safety enhancements in ArduHeli 3.4 and 3.5. But I don’t recall if this stuff ever made it to master, or into the mainstream code?

I think this is definitely related to the RC thing on rebind. But we question whether we should put patches in ArduPilot for what is obviously a defect in the RC or RC firmware.

[bnsgeyer (Bill Geyer)]

@ChrisOlson

Yep, that’s why asked if this is a FrSky radio with OpenTX. It sometimes takes 4-5 frames during rebind from lost signal to get all the channels back.

Yes but that doesn’t explain why the interlock is not disabled since RC8IN is zero during the lost link. That only explains the rebinding issue.

@ChrisOlson

But I don’t recall if this stuff ever made it to master, or into the mainstream code?

Yes I remember these changes but I’m pretty certain they never made it in master.

[ChrisOlson]

@bnsgeyer

Yes but that doesn’t explain why the interlock is not disabled since RC8IN is zero during the lost link. That only explains the rebinding issue.

Because when you set the RC failsafe to no frames, I think that’s the way it’s supposed to work. At least that’s what it does with my Taranis. So with no frames the controller goes into failsafe and it keeps the throttle on by itself. No frames means everything goes to zero. This is supposed to be the most modern way to handle failsafe.

It didn’t trigger the “motor interlock” until it got RC signal back with normal on Channel 3, and no signal on Channel 8. So now the RC failsafe is cancelled, and we have the same thing as the pilot commanding throttle hold in flight. So it shut down the engine for a bit, then re-enabled it when Channel 8 came back. And then the ESC didn’t re-initialize.

With setting a RC failsafe value on Channel 3 for radios that don’t do “no frames”, then you have configure your RC to hold last value on Channel 8.

Like in the Taranis you can set a model to “no pulses” (kills everything), Hold (holds all channels at last value), or Custom (have to set each channel for what you want it to do).

No pulses (or no frames in ArduPilot) is supposed to protect from the receiver coming unhooked in flight. The other options won’t protect if the receiver comes unhooked.

[ChrisOlson]

@bnsgeyer this can also happen if you put your flight modes on a aux channel instead of mode. Since mode has priority over aux. So if you have your flight modes on channel 7, it can cause a flight mode switch on re-bind.

Other folks have noted this

Copter randomly gains altitude and drifts in Auto mode

Hi; I am looking for some advice from someone that knows more about the flight logs than I do. I have only experienced this twice now but was at the same site each time. I have flown many other sites between these two experiences with no problems. I want to make sure I don’t have a serious problem that is just starting to show its ugly face, and hopefully I can prevent a future problem. What is happening is when in auto mode during a mapping mission the copter will suddenly start to climb in al…

[ChrisOlson]

@bnsgeyer

Yes I remember these changes but I’m pretty certain they never made it in master.

@bnsgeyer I looked thru our code history for heli, and we definitely had Rob’s patch in ArduHeli 3.3.3

https://github.com/ChristopherOlson/ArduHeli/commit/b0911873fe14c2fc74eeedde369462602263407d

But this never was PR’d to Copter. And then the code changed so radically with 3.4 and on that the whole series of safety enhancements we did in 3.3 were either addressed by new code in 3.4, or never got carried forward.

That being said, I’ve flown plenty of auto flights with 3.5 with total loss of RC and never have had an issue with it. Flown with both my FS-i10 and Taranis X9D+.

[carlosanlley (Carlos Sanlley)]

@ChrisOlson

So the question is, is this a Taranis radio?

I’m using a Spektrum satellite receiver connected directly to the Pixhawk to the spektrum port, the Tx is a DX8 and the binding process was done via Mission Planner.

@ChrisOlson

Most ESC’s want 1000 to 2000. So is your ESC calibrated for this 1100 to 1900 throttle range?

Yes, my ESC is calibrated to this range and was not set for autorotation.

@ChrisOlson

I also notice that normal range for RC8 IN should be the full range of your RC cal on channel 8. Your cal is 1099 to 1901

My default endpoints in the Spektrum DX8 are 1100 to 1900, I prefer to do all changes or setup in the Ardupilot and leave the Tx to stock values.

@ChrisOlson

But the signal only goes to 1500:

I’m using a Castle Creation in governor mode, 1500 is set to target a specific headspeed.

@ChrisOlson

So this brings up a question about having inadvertent mixes in the radio that may be causing a problem with why the throttle signal did not come back at the same time as the collective signal when the radio failsafe was cancelled upon re-bind.

I double checked and the Tx has all the parameters set to its defauld, no mixes are active in the radio, everything was configured in the Ardupilot. When the signal was lost, I see all RC Inputs going out at the same time as well as coming up at the same time, please see graph below. That indicates me that the signal was completly lost and no other action was made from the Rx, if not, we should see the RC IN going to that specific value (for example if the Thr failsafe in the Rx was set to Idle, then we should see the Ch 8 going from 1500 to 1100 in the RC IN values, am I correct?)

[carlosanlley (Carlos Sanlley)]

@ChrisOlson

Now the ESC made things worse by not re-initializing (possibly due to range calibration problem, autorotation window set, etc.).

I completely agree that if the autorotation windows was set to enable, in this particular case it might have prevent the crash, but if Tx signal loss continued for a couple of seconds more, it will have ended the same way.

[ChrisOlson]

@carlosanlley

When the signal was lost, I see all RC Inputs going out at the same time as well as coming up at the same time, please see graph below.

No, they didn’t all come back at the same time. Where the RC failsafe cleared you got the Channel 3 signal back. But RC8 remained at zero for several seconds.

So at this point the controller does exactly what the RC tells it to - there is no RC failsafe, there is no throttle signal - shut the engine down. This is a radio problem.

[ChrisOlson]

@carlosanlley

I completely agree that if the autorotation windows was set to enable, in this particular case it might have prevent the crash, but if Tx signal loss continued for a couple of seconds more, it will have ended the same way.

If you intend to fly BVLOS and hobby grade RC range on a regular basis I think I would recommend purchasing a more robust RC system for long range links. In the US anyway, BVLOS must be approved by the FAA for commercial operators and is illegal for hobby/recreation. For commercial the FAA requires that the pilot retain complete control of the aircraft at all times, including being able to demonstrate avoidance of manned aircraft when operating BVLOS.

There is various long-range RC systems available but all them require licensing.

[carlosanlley (Carlos Sanlley)]

@ChrisOlson

No, they didn’t all come back at the same time. Where the RC failsafe cleared you got the Channel 3 signal back. But RC8 remained at zero for several seconds.

You are right, looking it closer both times that the Ch 8 Out went down, Ch 3 In was up but Ch 8 didn’t.

Ch 8 was the only one did was not back on at the same time as the other channels, di you think that it might have something to do with Spektrum giving more priority to the lower channels?

[ChrisOlson]

@carlosanlley

Ch 8 was the only one did was not back on at the same time as the other channels, di you think that it might have something to do with Spektrum giving more priority to the lower channels?

I’m not totally sure, as I haven’t used a Spektrum radio for many years. I have a Spektrum DX-6i that I could test with a satellite receiver and see if I can figure out how it handles RC loss and rebind, looking at the channels on the scope. But it would be this weekend before I could get to that.

[carlosanlley (Carlos Sanlley)]

@ChrisOlson

BVLOS must be approved by the FAA for commercial operators and is illegal for hobby/recreation

I mainly do operations in Dominican Republic, here we must have license and registration with local aviation authorities and for BVLOS flights, special permits are needed were a box in the airspace is reserved for this specific UAV operation.

[ChrisOlson]

@carlosanlley

I mainly do operations in Dominican Republic, here we must have license and registration with local aviation authorities and for BVLOS flights, special permits are needed were a box in the airspace is reserved for this specific UAV operation.

The Dominican Republic has at least one 440 ham repeater hooked to Yaesu Wires II, as I’ve talked to operators there on it before on the 440. So that means that if you hold a ham license you could use a more robust system like DragonLink on the 440 ham band. I use DragonLink on my long range heli’s and with the antenna on a 15 ft poke it can communicate with the heli on the ground at 10km and 20km is no problem with the heli in the air.

Investing in a more robust RC system like that, IMO, is worth it for safety if you fly long range. It does require the licensing issues, but realize that hobby-grade equipment does have limitations.