Crash due to (SAFE) mode

Hi everyone,
we are using several Pixhawk 2 powered fixed-wing UAV (with a Here+ GNSS) for atmospheric research. Generally, the Cube performs very well and the handling and flight characteristics improved greatly compared to our old system. However, today we had the first crash. We flew a grid in 75 m relative altitude and everything went fine until suddenly the message (SAFE) appeared in the HUD and the plane started losing altitude. Switching to manual mode did nothing and the plane crashed because we couldn’t regain control. While reading the logs, I couldn’t find any reason for the crash. Can anybody here take a look at the logs and maybe help me find the reason for this behaviour?
the logfile

That should only ever happen with a hard reboot (full power cycle), or physically pressing/shorting of the switch. I can’t think of any other way for the safety switch to be enabled. I’d lean towards a hardware failure (not flight controller) - connectors is where I’d look first.

Question: did you have anything connected over USB?

I took a look at your log. It appears the log stops during your “grid” flying at 75m alt. (specifically, on the 3rd leg, during East-ward transit, about 2/3rds across the pattern) Is this when you saw the SAFE error and the un-recoverable descent began?

If yes, I am unsure if the dataflash log will have any clues or not. @james_pattison had a good idea with the electrical issues… maybe there was a switch-short or some other electrical problem?

The SAFE error occured at the moment the log stops, yes.
There was nothing connected via USB. Also, since the GPS is placed outside on top of the aircraft with nothing in the way, nothing can have pressed the button. It seems that a power cycle is a possibility, however I have disabled the safety button, so when powering up the aircraft, the servos move instantly (albeit not the motor). Is the (SAFE) mode different from the disarmed state after powering up? Sadly, the crash ripped the board the autopilot sits on apart, so checking the connectors may be a moot exercise…

I also discussed the crash in the Pixhawk 2 group at Facebook.
As Christian Mauch and Philip Rowse suggested, I will definitely change the setup of the power supply so that the Cube is powered by two BECs with a third BEC powering the servos (The current setup is an evolution of an older setup with a Pixhawk 1, where the backup power came from the servo rail). Also, I will use BRD_SAFETY_MASK so that all crucial servos can move in safe mode.

Here is the telemetry log from the flight: Telemetry log

On a standard setup, yes.
The (SAFE) mode indicates whether or not the safety switch keeps the servos from moving.
Arming/Disarming is separate, but also prevents the motor from spinning.
So they are separate, and either one (or both) can prevent the motor from spinning.

I’m not familiar with this… do you mean you have disabled the safety-button pre-arm check? Or something else?

something went very bad, the dataflash log ends some 20 seconds before where the .tlog ends, I do not see a reboot, so it was not that.
tlog:
your BRD_SAFETYENABLE=1 , so your safety switch was on, could it have been damaged/bad/wet or otherwise disarm it ? - it looks like that.
even your HWSTATUS.Vcc confirms the disarmed state.

Phil Rowse and Tridge are looking into this (on a Facebook thread), so I think it’s getting the best assistance possible.

Deja-Vu!

I had a crash months ago, that had a very very similar pattern. See the thread linked below -

https://discuss.ardupilot.org/t/possible-crash-due-to-pixhawk-failure-need-help/17710/2

Would love to re-open this case and get answers to some of the questions

Hi Nihal, I have linked your thread in the facebook discussion.

I will continue to also post updates on this case here and provide the solution, if I find one.

The current status is that it looks like the crash occured due to a short in the wires leading to the safety switch or a malfunction of the safety switch itself, which enabled the safety mode. Andrew Tridgell and Philip Rowse had a look at the logs, and it looks like the only way safety mode could be enabled in my case was by an activation of the button.

I installed a custom cable connection (servo plug) between the safety switch and the pixhawk. Maybe that connection shorted, I will investigate that. In the meantime, Philip Rowse was so kind to offer to have a look at the Cube and the Here+ unit.

Hopefully we will find the reason for this crash. Some further information on the flight: It was quite cold (4°C on the ground) and very humid (albeit not raining). There were almost no turbulences and a constant wind of 5 m/s, so the flight was very smooth.

(SOLVED)
After sending the crashed unit back to Philip Rowse​, he had a look at it and found nothing wrong with the Here+ and the Cube.
His (and also my) preferred explanation for the crash is that due to the humid conditions on the day of the crash water condensed inside the Here+ unit, leading to a short circuit in the safety switch. This enabled the SAFE mode in Arduplane which disabled all controls as well as the motor, which in turn led to the loss of altitude and unwanted contact with a potato field.
My workaround for this issue is setting BRD_SAFETY_MASK so that all needed channels are still working in SAFE mode and change BRD_SAFETYENABLE to 0 so that all servos can move immediately after boot. The safety issues that arise from this change have been addressed in our checklists. The UAV is remotely armed after attaching it to the winch and moving away from it and disarmed before any personnel on the ground can pick it up.
Anyway, a rugged version of the Here+ would be nice :wink: