Cumulation of crashes due to seemingly weird input (Pixhawk 4)

sure, you use the BRD_SBUS_OUT option to setup SBUS output on the SBUS output port of the board. I’m using a Pixhawk1 as the generator.
Then I added this little hackish snippet of code in ArduPlane/servos.cpp in Plane::set_servos:

    float t = AP_HAL::millis() * 1.0e-3;
    for (uint8_t i=0; i<8; i++) {
        SRV_Channels::set_output_pwm(SRV_Channels::get_motor_function(i), 1500 + sinf(t*(1.0+0.1*i))*250);
    }

then I set SERVO1_FUNCTION to 33, SERVO2_FUNCTION to 34 etc. The result is a sinusoidal output like this:

that is plugged into the RC input of the Pixhawk4. It simulates moving the sticks rapidly in a very recognisable pattern. It produced the issue with the Copter-3.6.2 stable release on a Pixhawk4 in a few seconds. I’ve been running it for about 4 hours now without an error with the new firmware.
Cheers, Tridge

thanks @tridge I am generating sbus message with ESP https://github.com/Huibean/sbus-generator/blob/master/sbus-generator.ino and testing it on pixhawk4 and will give you some feedback later, by the way, can you show your pr link?

great, sounds good. Note that I found that the issue came up more with lower frame rates of SBUS. I don’t know why that is, except to think that it is related to duty cycle at idle voltage. I noticed the tool you are using has a high update rate (8ms), which means it probably won’t cause the issue as quickly.
PR here:

1 Like

I’d really appreciate it if someone could do a test flight with these changes. I don’t have a frame setup with a Pixhawk4.
Firmwares are here:
http://uav.tridgell.net/Pixhawk4/
once it has been flight tested we will push these into the stable firmware folder for all Pixhawk4 users.
Cheers, Tridge

Thanks for your work. I hope I will find some time on the weekend.

As far as I know, FrSky (e.g. the X8R) sends S-Bus frames every 10 ms.

@tridge : You wrote, that you are working with “Arthur from Holybro” on this issue. In post #4 you mentioned, that the IOMCU (a STM32F100) is responsible for SBus parsing and that the firmware is identical to the one used for copter stable releases running NuttX (and will be changed to ChibiOS based with copter 3.7.x).

What I do not understand: Why is the issue only related to Pixhawk 4? Which role has the hardware particularly in this case? If the issue is related to STM32F100 I/O processor, would’nt this mean, that also Pixhawk 1, Pixhawk 2, Pixhack v5, which also use the STM32F100 are affected in the same way - and already with earlier copter releases? Is the HAL code for the STM32F103 (used in Cube 2.1) different?

Good evening,
Has anyone tried to reproduce the issue with Pixhawk 4 + PX4?

Re px4, I think this was the same root issue they saw here: https://github.com/PX4/Firmware/issues/9595 and https://github.com/PX4/Firmware/issues/8263
@dagar

I’ve discussed this with Arthur, and it seems like that the difference is in the way that the DSM and SBUS input pins are arranged. On the Pixhawk4 the DSM and SBUS pins connect to the same external pin, with an inverter on the SBUS pin. This is different from other boards, which have the DSM pin entirely separated.
My guess is that the enabling of the DSM pin for RX gives a small pullup on the SBUS receiver, which affects the integrity of the signal. I’m going to experiment with changing the GPIO setting on the DSM RX pin to see if it makes a difference. We have a fix now, but it would be nice to be able to get the pin setup to the point where the protections I’ve added to SBUS parsing are never triggered.
Cheers, Tridge

Don’t know, if this helps: I measured that the SBus RX pin has “high” level (3.27 V), when not connected. When it is connected through a 12 kOhm resistor to ground, voltage goes down to 0.75 V, which means there is an internal pullup of about 40 kOhm.

@tridge On the AP what is the init state of PG14 USART6 TX and is it initialized to 1 wire mode?

On FMUv5 (pixhawk4) There is a wire-or with 220 Ohm resistors.

F7 PG14 TX-----------\/\/\/-----------±----- SUBS IN J8-2
F1 PA10 RX-----------\/\/\/\/-----------+

This means that a) The F7 must either leave the PG14 in the HW default (input) OR b) init to 1-wire mode.

1 Like

Why was v3.6 released as a stable version if the issue was known?

Hi David, thanks for dropping in!

not by default, no. We do have SERIALn_OPTIONS parameter bits in master to enable RXINV, TXINV, SWAP and HDSEL, but for now we don’t turn any of those on by default on FMUv5.
You’re absolutely right that I missed the connection to PG14, and that PG14 being configured by default as USART6_TX gives it a pullup. I’ve tried changing that, and it does get rid of the pullup, but doesn’t actually improve reliability. I still find that if I use the old NuttX based IOMCU firmware that I get occasional SBUS input corruption. It is less frequent, but still happens. I don’t get any corruption with the firmware builds I linked above that switch to the ChibiOS based IO firmware.
Still, I agree that configuring PG14 as either an input or 1-wire is the right thing to do, even if it isn’t a complete soln.
One theory I have for why the NuttX iofw in older ArduPilot builds has issues is that it may lose RX interrupts on PB11 (USART3_RX) on the F1. Perhaps the default of -Os is increasing interrupt latency enough to be losing bytes, probably in the DMA callbacks for the USART2. Under our ChibiOS builds we compile the DMA callback code with -O3 to keep interrupt latency low, but we don’t do that for the old NuttX build.
That wouldn’t explain why this is different from what happens with the Pixhawk1 and Pixhawk2 though.
Which leaves just one other key difference, the one Arthur pointed out to me, which is that on the Pixhawk4 the DSM input on F1 PA10 is the same external pin as the inverted PB11 for SBUS, whereas on Pixhawk1 and Pixhawk2 the DSM input is separated. So maybe when we lock onto SBUS we should change PA10 to be an input.
Either way, the fix I’ve linked above does work reliably. I’ve run it for many hours with no issues with the above SBUS stress tester input.
Cheers, Tridge

I didn’t know about the issue at the time of release. I had done quite a bit of bench testing of my two Pixhawk4 boards and hadn’t seen the issue. Once I saw this thread I tried a lot harder to reproduce, and found I could reproduce with a SBUS stress test.
You are right that we didn’t get onto this as fast as we should have though, and I apologise for that.
Cheers, Tridge

Tridge & David: do I understand it right, that the SBus Pin PB11 has a connection to PA10 (DSM) as well as to PG14? To me that sounds a bit confusing and it increases the risk for GPIO configuration errors. If someone mods the hardware by seperating these lines that may carry another risk that there is no pullup anymore on the SBus input which means it becomes free floating. If the receiver fails, you may also get this rubbish on the input.

I lifted PA10 off the board, as I don’t own or use DSM. Signal now runs only thru the SBUS inverter ( 8-pin chip marked V240 ) to PB11.

For those like me wondering if this is released, yes, this fix is in copter 3.6.3

1 Like

Summarized, I still do not see the “fix” as an elimination of the problem at the root cause. You may be able to detect irregular bytes by checking the parity bit - but what happens in that case? We see from the logs of the crashed drones that there is not only a wrong packet sporadically that could easily be dropped. When the problem occurred, the logs recorded weird values on the channels continously until the drone crashed. If the “fix” means, the controller keeps using the last valid channel value, this would mean the drone will still be out of control.

Looking at other FMUv5 controllers (Pixhack V5, Pixhawk 4 Mini), they also share DSM and S-Bus input on the same pin - but are they also connected to USART6 TX or is this a particular difference for the Pixhawk 4 (and propably the root cause)?

For me, the situation is still not cleared and this means I will not trust Pixhawk 4 until I see something official from Holybro stating that the root cause is eliminated instead of calming the symptoms.

1 Like

Could someone, who made a hardware mod (seperating the S-Bus input) show a picture or drawing, please?

1 Like

I plan to rebuilt my crashed copter in a few weeks. Can anyone confirm wether or not it’s safe to use the pix 4 currently? Is the problem truely solved?

Please answer honestly, I’ll rather buy a DJI controller than crashing my expensive and heavy copter once again.