Follow up work from this post , we decided to go for another flight to get a better tuned pitch controller, rectify airspeed sensor problems and collect more flight data before we push for TECS tuning.
We had 1 successful flight before this. Good thing is we have logs from 2 flights: 1 that flew ok. 1 that ended in a painful crash.
For this flight, we took off and had reasonable QLOITER performance. I then switched mode from QLOITER to AUTOTUNE @ 20 meters. We saw a good transition. However, just when transition is declared done, the aircraft, with plenty of airspeed, decided to take a nose dive over a period of ~1.5 to 2 seconds.
We have been studying the logs over the past 2 days and have not been able to identify what is happening.
Points from on hardware that may not show on the dataflash logs
Post crash, Cube and servo power supply components still work.
Post crash, progressive build up of tests on servo system: all servos still work with servo tester, CAT6 cable + booster board pairs we got from ServoCity and when connected to MAIN2 of the smashed out Cube (apparently still working) in FBWA mode.
Pre crash, CG was checked to be nose heavy prior to flight. It was well below it’s design MTOW limits.
Pre crash, we did not do an accelerometer or compass calibration after the last flight, which was 1 month ago.
Pre & post crash, FBWA direction of PID compensation was checked to be correct.
Key timings of this Flight
12:02:26 = Transition begins
12:02:34 = Dive begins
12:02:37 = Last data
A few hypotheses that we have been thinking about:
Vibrations from the DLE61 gas engine got to the Cube’s pitch estimation, causing the dive
Looking at the attitude graphs, the system knew that pitch was going down. It was also desiring more and more pitch as the aircraft dove downwards.
error_RP data from NKF4 seem to indicate small errors.
Plane pitch controllers (set of PIDP data) also seem to indicate that PID loops are actively kicking in to bring aircraft back to level
One thing we did notice is that there was a lot of clipping of accelerometers while aircraft is still on the ground. However, the clipping events stopped when aircraft is in the air.
Given that we were able to do a QLOITER with an engine in idle @ 2700 RPM, we aren’t sure if sensors and estimation are the cause of the problem.
GPS came loose and fell off right after transition is done
No abrupt changes to magnetic heading and GPS lock status.
Servo system failed & servo linkage arm broke in flight, right after transition is done.
Post crash hardware tests seem to indicate otherwise. Because it was a nose dive, most of the rear of the aircraft was salvaged and test was possible.
Post crash mechanical checks indicated that linkage arm and servo horn was still attached.
Vibration matched resonance frequency of some component within the Cube, causing the Cube to send wrong signal out to MAIN2.
We are not sure how to verify this
Auto trimming of servo acted in the wrong way, coupled with effects of bad vibration on pitch estimation , causing elevator to compensate in the wrong direction.
We are also not sure how to verify this
Can we request some help from this community in evaluating the logs along with us (i.e. new ideas or hypothesis) We are still looking at the data ourselves but it is getting to a point where we’re running out of ideas.
Could your elevator be connected backwards? Once the transition is done the quad motors switch off, during the transition the quad motors would overpower the elevator and mask the issue. This would also be true if the linkage was broken.
You could verify in the logs if the front motors are having to throttle up more than the rear motors as the airspeed increases.
I don’t think any of the other your hypotheses would cause such a dramatic crash.
If you have it working nice in Qmodes you should look at setting up Qassist, if correctly setup the Quad motors will come on if too large a angle error is detected, This may have been able to catch it in the nose dive.
I’ll vouch for q_assist. it can be a huge help and has saved our aircraft or greatly lessened the damage many times.
When in AUTO modes (MC or Plane)I also always keep my throttle up at 50% and Qloiter as one of my available flight modes so I can switch back at a moments notice. This has also saved us many times, and can allow you to detransition your aircraft back into MC mode at anytime in case something funky happens in AUTO plane flight.
hi TC,
I had a look at both your older log (03_Nemo1stAutotuneFlight.BIN) and the new one (00000003.BIN). I can’t find any reason for the crash with the new flight. The elevator output looks right, the servo direction settings are the same, the sensors look consistent. There is significant vibration, but I can’t see any way that the vibration could have caused this failure.
So it looks like a hardware failure, for example a failed servo linkage for the elevator. If Q_ASSIST_SPEED had been enabled then it would have had a reasonable chance of saving the aircraft, although I know that isn’t much consolation now.
Going through your analysis:
this definitely was not the cause, as the 3 attitude estimators (2 EKFs and 1 DCM) all agree.
there is no evidence of that in the sensor logs, and even if it happened it wouldn’t cause this.
this type of failure seems the most likely given the log. Could a servo cable have come loose? Are the servos powered from the servo rail of the flight controller? I notice the servo rail is measuring 7 volts. I presume that is deliberate? I don’t see any change in servo voltage associated with the failure, so no smoking gun, but looking at power and cabling for the servos is an obvious candidate.
The raw servo output logs show it wasn’t an auto-trim issue. The trim on the elevator is the same in this flight as your earlier successful flight.
Thank you for spending time looking at our logs. It does help to confirm some hunches that we have about what went wrong.
47 Hz featured quite prominently in the FFT. It was close to the idle RPM that we had that day. The system was probably exposed to it for 20 seconds while still on the ground.
Another hypothesis we have is that the vibrations - oscillating between 5g to 9g on VibeZ - hit the resonance frequency of the servo that we are using, causing it to jam in flight
We’re hoping we can replicate the servo failure by running engine tests on the ground.
Well, at least now we know how a hardware failure can look like in the logs.
That is why larger planes tend to have dual elevators, so you can lose one. Each elevator should be on a different servo output pin on the autopilot (so separate cabling).
Also, enable the Q_ASSIST_* settings. They are very worthwhile.
Cheers, Tridge
I am still trying to hunt down what could possibly cause this. As I mulled over it the last few weeks, it dawned on me that
Our DLE 61 engine was not grounded to the system’s power ground.
When installing an EFI on a DLE170 on previous project, I used to connect one of the engine’s fins to the battery ground. However, on the airplane that crashed, because it was on a carb, it didn’t cross my mind to also do it.
The hypothesis is then: because the engine had no connection to ground, the engine built up charge and released it into the system (possibly via the ignition), causing something inside the Pixhawk 2.1 to transiently behave really badly
Something else that we are also grappling: why is it that we didn’t see anything amiss with our servos during our 1 week of post-integration shake down with the engine (where we put the system through its paces by revving the engine and running it for long hours) on this system when it was integrated? Could it be that the charge was discharging to the ground?
It seems like the only was I could “try” to “replicate” this is to “shoot” the system down with a ESD gun
@tridge Is the engine of Canberra UAV’s OctoPorter connected to the battery ground (i.e system ground?)
Additionally, on another note: can I also ask if your team are still using a fuel pump with the engine and if so, which one are you guys using?
