Cumulation of crashes due to seemingly weird input (Pixhawk 4)

ArduCopter Copter 3.6
1

Back in August, using AC 3.6.0 RC7 on HolyBro’s Pixhawk 4 I had a weird crash on the country side in northern Germany. After some successfull flights, during another flight I suddenly lost control over the copter. It started doing weird things and finally drifted into a tree. Looking into the log, I saw weird signal changes on all channels, also the flight mode was rapidly changing:


https://farm2.staticflickr.com/1876/42445673810_e880d24515_b.jpg

First, I thought a nearby (1,5 km) radio station (sporadically sending wheather reports) was the originator. I was using a Taranis X9D+ with X8R reciever and 433 MHz telemetry link.

The strange behavior could be reproduced at that location also on later tests (only on ground without propellers). I had activated the Taranis log in parallel and there everything looked normal. The Pixhawk log again showed rapidly changing RC input and rapidly switching of flight modes:


https://farm2.staticflickr.com/1878/44476204211_15aae5a980_b.jpg

Meanwhile another user of Pixhawk 4 and AC 3.6.1 (stable) had a crash in Switzerland with exactly the same pattern. He also lost control over the copter and after weird movements it crashed. He was also using Taranis X9D+ and X8R receiver. In his log you can also see these strange RC input patterns and the rapid mode switches:


https://farm5.staticflickr.com/4839/32158158628_02ca95828f_b.jpg

After some further research, we found another report with the same weird behaviour:

This user was using Pixhawk 4 with Taranis X9D+ and R9M + R9 receiver.

Meanwhile I do not believe anymore that this strange behaviour was induced by the receiver. In my examples you see that also flight mode “SIMPLE” was activated although I did not assign it to any RC input.

My guess is that there is either a bug in AC 3.6.0/3.6.1 (ChibiOS) or a hardware bug on the Pixhawk 4.

2

Do you have simple mode assigned to any of the flight modes using the simple mode bitmask? Or do you not use and have no assigned any usage of simple mode at all in any of the parameters? Also, how do you have your receiver connected to the autopilot?

3

Helge,

Thanks for the report. Do you have a dataflash log by any chance?

4

Thanks for the report. I have two Pixhawk4 boards here and I haven’t seen the issue, so I’d like to get some more information from you that may help me to reproduce.
What binding mode are you using with the X8R? How many channels are being transmitted? How often are you able to reproduce the problem? Is there a pattern of stick movement that produces the problem more quickly?
I’d like to get your full parameters, plus a log and also a copy of your taranis model file if I can so I can load it onto my taranis.
One thing I would note is that it very hard to see how it could be a ChibiOS problem, as those copter releases do not use ChibiOS on the IOMCU. The IOMCU is a STM32F100 that does all the “main” output and is also responsible for RC input (including parsing of the SBUS signal that the X8R will be producing). The firmware running on the IOMCU is identical to the one used for copter stable releases running NuttX.
In master and for the next major release of copter (the 3.7.x release) we have changed that IOMCU firmware to be based on ChibiOS, and we have rewritten the SBUS parsing code. That does in fact make SBUS parsing more robust as it adds parity checking on the UART (a feature that NuttX doesn’t support).
Either way, if you can get me the above details I will try to reproduce. I’d particularly appreciate you trying to find a way to trigger the issue quickly.
Cheers, Tridge

5

Hello,
I’m the one that had a similar issue flying in Switzerland.
Here’s the log from my flight:


Unfortunately I have no access to the copter and other stuff until next weekend, so I can’t post a .bin file or a taranis file.

I was using the X8R in the D16 mode, I think mode 3 to be more specific. I was using this same binding mode on a different copter (using an auav-x2) for more than 3 years without running into any issue.

The copter flew very well for a few hours before the incident. I have to mention that it was my first time flying at the place of the crash. In case it’s relevant: radio/tv etc. transmitter in Switzerland can be found here: https://map.geo.admin.ch/?topic=funksender&lang=de&bgLayer=ch.swisstopo.pixelkarte-farbe&layers=ch.bakom.radio-fernsehsender,ch.bakom.mobil-antennenstandorte-gsm,ch.bakom.mobil-antennenstandorte-umts,ch.bakom.mobil-antennenstandorte-lte,ch.bakom.richtfunkverbindungen&catalogNodes=403,408&E=2661400.00&N=1190600.00&zoom=3

I’d also like to mention that the crash detection was a bit slow. The copter was already on the ground (lying on it’s back) when some of the motors went (nearly) full throttle again, causing even more damage.

6

Hello
I had similar problems a long time ago, with the Taranis the 3 way switch was broken, so that the flight modes were switched very quickly. The transmitter had switched off, the copter flew back to falsafe.

7

Thanks for your involvement. Here are links to three logfiles:

https://kopterkraft.com/downloads-static/2018-08-23%2016-42-10.zip
https://kopterkraft.com/downloads-static/2018-09-02%2018-45-54.zip
https://kopterkraft.com/downloads-static/2018-09-02%2018-57-38.zip

The log from 23 August was the flight that crashed. Before the crash, I tested some waypoint mission flying at high speeds (which ran into an EKF error propably due to high Z-vibes) and after a short landing I carried out an autotune. During that autotune I did not make any stick movements and I had done already several successfull autotunings before with that drone and AC 3.6.0RC7. Suddenly the drone got this mad behaviour.

The logs from 02 September were recorded on the same location (since I thought, that there may be an influence by the nearby wheather radio station). This time, I armed the drone only on ground and without propellers. The first of the two logs showed that strange behaviour right from the beginning. In the Taranis log that I recorded in parallel, everything was normal. The second log from that evening showed that strange behaviour again after 73 minutes. Most of the time, the sticks on the Taranis were untouched, only sporadically I moved them just to check, if there is still control.

@Pedals2Paddles: I have done nothing “willfull” in the configuration that would assign “SIMPLE” mode to any RC channel. I assume the bin files contain also the configuration.

@rmackay9: See the links to the logs above.

@tridge: The X8R was connected via S-Bus with the cable coming with the Pixhawk 4. The Taranis was set to D16 mode, channel range from 1-16, failsafe mode was set to “receiver” and on receiver, failsafe was configured to “no pulses”. The failsafe tests were successfull.

8

Hello Rainer,
In our cases we had weird amplitudes on all channels, not only the 3 way switch, so I am pretty sure this is not the problem.

On the german ‘Kopterforum’ Helge said that he once used the logging function of the taranis when the issue happened and apparently there were no weird amplitudes in the taranis log, so the problem has to be somewhere on the copter.

9

Hi,
The following are the files from my crash
kmz file
log file
gpx file
kml file
param file

HTH,
Gal

10

The most important thing is if anyone can find a way to reproduce it on the ground. Has anyone who has seen the problem tried to reproduce on the ground?
Cheers, Tridge

1 Like
11

Hello Gal,
did you check if there were any significant radio stations around the area where you crashed?

12

@tridge, as you might have seen in my post 1h ago, I could reproduce it on ground but until now only in the area where I had the crash and it happens only sporadically.

1 Like
13

Anyway, not depending on whether the issue is caused by the taranis, X8r, pixhawk 4 or Ac3.6 code - wouldn’t it be possible to implement a failsafe to prevent such crashes?
Possible solutions could be

  1. if there are more than x (e.g. 10) flight mode changes within y (e.g.10) seconds, the copter goes into RTL, lands, disarms and until this is accomplished it ignores all rc inputs, except if a certain stick-combination is used (and using this combination would imply that the rc signals are now working properly)
  2. If there is a signal on a specified channel (e.g. channel 10) that is not actually in use, that differs from a certain range (e.g. 1450-1550) then the copter goes into RTL (while also ignoring all rc inputs except if a certain stick combination is used).

I think especially the 2nd solution might actually make sense (in multiple scenarios). What do you think?

1 Like
14

Just to eliminate potential reasons: The pullup resistor (about 40 kOhm internally) on the signal wire of the S-Bus input is activated as expected. This means there is no “floating” input in case the receiver fails.

1 Like
15

I think in cases like this it is much better to go to the bottom of the problem and not just hide it. Reason is that if it is a bug you never know what could be involved and how it’ll show up next time.

Corrado

16

channel num issue,see here:

image
image.jpg1578×842 152 KB

1 Like
17

@tridge this is a log reappear on the ground https://drive.google.com/open?id=1CqcjHzAwFsNrby2oq4t-OJzCakYoT6RK
The timestamp around 60000, channel 1-6 output under regular pattern, and not paired chan 7-14 has values seems then

image
image.png1579×622 336 KB

image
image.png1579×622 237 KB

This commit suppose to fix this, but this issue still appear in 3.6.2

transmit : Futaba T8FG,
version: Copter 3.6.0,
reapear way: fast moving sticks for a while, not sure the exact causing fact yet

1 Like
18

Hi @Hacky ,

Actually there is a huge radio station about 1-2 km from where it happen.

My logic ruled it out as the source of problem since, “how is it possible that an outside source which is not bound to my receiver send commands?” doesn’t make sense to me.

It seems that there are two constants in this equation, Pixhawk 4 and ChibiOS in which the same code of ChibiOS runs on other FCs without a problem, makes me wonder.
Gal.

1 Like
19

@Gal, my assumption was, that not the receiver induces the “noise” but that the flightcontroller hardware itself uses a design where signal lines act as an antenna. Pulsed long wave radio transmission could be misinterpreted as commands. But there would still be the question, why I see flight modes which were not assigned in the RC configuration.

1 Like
20

Thanks @Huibean - opening it with the mission planner logviewer shows a very similar pattern, indicating the same problem:

45158533655_e233c5a9f7_b.jpg1023×481

Good to know that this happens also with other RC. I do not think, tha fast moving sticks are the cause. In my cases there was almost no stick movement.