Cumulation of crashes due to seemingly weird input (Pixhawk 4)

Hi @Hacky ,

Actually there is a huge radio station about 1-2 km from where it happen.

My logic ruled it out as the source of problem since, “how is it possible that an outside source which is not bound to my receiver send commands?” doesn’t make sense to me.

It seems that there are two constants in this equation, Pixhawk 4 and ChibiOS in which the same code of ChibiOS runs on other FCs without a problem, makes me wonder.
Gal.

1 Like

@Gal, my assumption was, that not the receiver induces the “noise” but that the flightcontroller hardware itself uses a design where signal lines act as an antenna. Pulsed long wave radio transmission could be misinterpreted as commands. But there would still be the question, why I see flight modes which were not assigned in the RC configuration.

1 Like

Thanks @Huibean - opening it with the mission planner logviewer shows a very similar pattern, indicating the same problem:

Good to know that this happens also with other RC. I do not think, tha fast moving sticks are the cause. In my cases there was almost no stick movement.

From Christian’s log, there seems to be be on more observation:
At RC12, spikes already appeared from the beginning, and at RC10 and RC11 the spikes are from exactly mid up to full (on RC10), but never lower than mid. Respectively from exactly mid down to low (on RC11), but never higher than mid. It seems to me like a quite artifical pattern, not like some wild turbulence.

Hi,
@fingadar Ch12 is the trigger for the IR-shutter that I’m using for the sony nex 5t. So those spikes are on purpose.

The phenomenon that the spikes only go in one direction can be observed in Helge’s log, too.

@anon67614380 I agree. But in case it’s a hardware issue with the pixhawk 4, it might be the only solution? Also, such a failsafe might be able to prevent different issues aswell?

@gnitzan I think the radio station thing is something we should certainly look further into. In my case, there was a radio station rather close aswell (my copter flew well for a few hours before the incident happened, but all the flights before tha fatal one were further away from radio stations).

I have now managed to reproduce the problem with Copter-3.6.2 stable on a Pixhawk4.
The problem does not happen with the ChibiOS based IO firmware. I will see if I can bring a fix into the stable release.

@all : The dealer, where I bought the Pixhawk 4, now got a fixed firmware from Holybro based on Arductopter 3.6.1. Seems like they warned regarding a problem when reading S-Bus with Arducopter already weeks ago. :frowning:

@tridge I got a file ending with “.apj”, size 828 KB, so this is probably already the binary build. Would it help you to compare, if the fix adresses the same part as your fix, when I send you this file?

Alternatively, Holybro should publish the source in order to bring it into the master, as - from my understanding - the license model of Arducopter would require this.

Do I understand that right: Holybro knows (and knew) that there is a major issue but didn’t bother to inform its customers?

1 Like

Same problem on ArduPlane Stable version and Dev Version.
I have had the same issue on a test run at the runway.
First also tought it is a issue with the S.BUS receiver.
I use a FrSky Hourus X12S and the X8R receiver. All signals by S.BUS.
Attached the log file from that test.

Best regards,
Markus

I’ve now got a very fast way of reproducing the problem by generating an SBUS output stream from another flight board and producing a rapidly oscillating output pattern. It produces the problem within a few seconds with the current stable ArduPilot releases.
I’ve got a fix for it and have built firmwares for users to test here:
http://uav.tridgell.net/Pixhawk4/
I’d appreciate it if as many people as possible with a Pixhawk4 test this and report any success/failure.
What the fix does is switch the IOMCU firmware to using ChibiOS instead of NuttX. ChibiOS has much better protections on SBUS as it is able to correctly detect parity errors. It also includes some improved framing error detections for SBUS input.
Please note that I am also working with Arthur from Holybro on this issue.
Cheers, Tridge

1 Like

Hi @tridge, can you show some detail on how to generate the SBUS output stream?

sure, you use the BRD_SBUS_OUT option to setup SBUS output on the SBUS output port of the board. I’m using a Pixhawk1 as the generator.
Then I added this little hackish snippet of code in ArduPlane/servos.cpp in Plane::set_servos:

    float t = AP_HAL::millis() * 1.0e-3;
    for (uint8_t i=0; i<8; i++) {
        SRV_Channels::set_output_pwm(SRV_Channels::get_motor_function(i), 1500 + sinf(t*(1.0+0.1*i))*250);
    }

then I set SERVO1_FUNCTION to 33, SERVO2_FUNCTION to 34 etc. The result is a sinusoidal output like this:

that is plugged into the RC input of the Pixhawk4. It simulates moving the sticks rapidly in a very recognisable pattern. It produced the issue with the Copter-3.6.2 stable release on a Pixhawk4 in a few seconds. I’ve been running it for about 4 hours now without an error with the new firmware.
Cheers, Tridge

thanks @tridge I am generating sbus message with ESP https://github.com/Huibean/sbus-generator/blob/master/sbus-generator.ino and testing it on pixhawk4 and will give you some feedback later, by the way, can you show your pr link?

great, sounds good. Note that I found that the issue came up more with lower frame rates of SBUS. I don’t know why that is, except to think that it is related to duty cycle at idle voltage. I noticed the tool you are using has a high update rate (8ms), which means it probably won’t cause the issue as quickly.
PR here:

1 Like

I’d really appreciate it if someone could do a test flight with these changes. I don’t have a frame setup with a Pixhawk4.
Firmwares are here:
http://uav.tridgell.net/Pixhawk4/
once it has been flight tested we will push these into the stable firmware folder for all Pixhawk4 users.
Cheers, Tridge

Thanks for your work. I hope I will find some time on the weekend.

As far as I know, FrSky (e.g. the X8R) sends S-Bus frames every 10 ms.

@tridge : You wrote, that you are working with “Arthur from Holybro” on this issue. In post #4 you mentioned, that the IOMCU (a STM32F100) is responsible for SBus parsing and that the firmware is identical to the one used for copter stable releases running NuttX (and will be changed to ChibiOS based with copter 3.7.x).

What I do not understand: Why is the issue only related to Pixhawk 4? Which role has the hardware particularly in this case? If the issue is related to STM32F100 I/O processor, would’nt this mean, that also Pixhawk 1, Pixhawk 2, Pixhack v5, which also use the STM32F100 are affected in the same way - and already with earlier copter releases? Is the HAL code for the STM32F103 (used in Cube 2.1) different?

Good evening,
Has anyone tried to reproduce the issue with Pixhawk 4 + PX4?

Re px4, I think this was the same root issue they saw here: https://github.com/PX4/Firmware/issues/9595 and https://github.com/PX4/Firmware/issues/8263
@dagar

I’ve discussed this with Arthur, and it seems like that the difference is in the way that the DSM and SBUS input pins are arranged. On the Pixhawk4 the DSM and SBUS pins connect to the same external pin, with an inverter on the SBUS pin. This is different from other boards, which have the DSM pin entirely separated.
My guess is that the enabling of the DSM pin for RX gives a small pullup on the SBUS receiver, which affects the integrity of the signal. I’m going to experiment with changing the GPIO setting on the DSM RX pin to see if it makes a difference. We have a fix now, but it would be nice to be able to get the pin setup to the point where the protections I’ve added to SBUS parsing are never triggered.
Cheers, Tridge