Plane did not respond to my inputs while in FBWA mode as you can see, it appears to have just locked its control surfaces. I’ve flown this plane on dozens of flights in FBWA, loiter, RTL, and takeoff modes and never had issues … till today.
LOG FILE (link)
… at least it was low tide!!
Sorry to see that. I have one of those sitting in the garage waiting for it’s maiden flight.
The FPV horizon seemed to be moving appropriately so that says the FC still “knew” where it was, even if it couldn’t or wouldn’t do anything about it.
I’m assuming you commanded the switch to FBWA? It didn’t just happen on it’s own? Did you try changing modes again to recover?
Unless you’ve cranked up your maximum angles then FBWA should have at least reduced the bank angle to max from the knife edge you were on. This would maybe rule out a radio issue.
Maybe start by looking at the servo outputs in the log. See if the aileron servo was trying to make the correction to roll to level. If the proper output was there, that could point to a servo or mechanical issue. If the output isn’t there or the output is holding the bank then it’s back to the FC.
Can’t say too much more without a log.
that is correct, I commanded it, I changed back at the last second as you can see and it just started to roll back over as I took command.
Looks like it did start to roll back like the video showed just before impact when I took over manual control, and you can see the servo outputs flipped as well.
Question is, why did FBWA not roll it back over?
In that graph, what output is what? Which is Aileron, which is Vtail-L, Vtail-R?
Edit, I"m guessing RC 2 is aileron. What is the value when it flatlines during the FBWA?
PARM, 182572003, SERVO2_FUNCTION, 4
PARM, 182572201, SERVO3_FUNCTION, 79
PARM, 182572480, SERVO4_FUNCTION, 80
So
C2 = aileron
C3 = left V-tail
C4 = right V-tail
ref1 ref2
Would you be willing to post a parameter file? Save me asking a million questions. 
What is the value of C2 when it flatlines?
What is the SERVO2_TRIM?
Is there any input or graph on the aileron input channel?
Sorry was in the processing of trimming it. I’ve cut a irrelevant section from line 4707 onward.
Sorry, I’ve been staring at this log and I can’t see any smoking gun. I’m not saying the answer isn’t there, but I can’t see it.
The fact you were in FBWA for almost a minute flying around just fine before you switched to Manual says that all the basics of the plane and the setup must be functional. Between the video and the log, the plane looked like it was flying well in FBWA.
When you reverted back to FBWA after manual, it’s like something just stopped. There’s a ripple in the RC2 output so it’s not “dead” but there just isn’t enough to do anything. There is activity in the PID values, and the desired roll seems right. I just can’t explain why it didn’t move the servo. This might suggest something deeper in the flight controller. That’s troubleshooting above my pay-grade.
Thanks for taking the time to look. I may setup a SITL test and see if I can reproduce the issue. That’s what I’m worried about is that maybe there’s an issue in the logic somewhere.
May have to try reproduce this IRL once I get it flying while being prepared for this behavior.
I’m not sure how it hit the ground, but rolled through level. I’ll bet if you’re ready on the manual mode, if this happens again you could recover it. That would be excellent data for the devs, if they knew that for sure.
Just goes to show how having an easy reach manual mode switch can be important. For me it is one of the 6 position hence took a while to reach for. Initially I thought I had stalled the wing, and tried to push down and go with the roll (at least, I think I did) then when it wouldn’t pull up did I realize I had some real issues.
Prior to this I’ve tried to push this plane into a stall many times while at high altitudes in manual mode and it’s always been very tame which is why I was confident enough to throwing it around in manual as I did.
Time to find a ultrasonic cleaning service, and get some de-ionized water clean the pcb’s and see how much gear is still functional.
I’ve rebuilt the plane and got it flying again.
Casualties:
None of the software settings have been changed other than moving some flight mode switches around.
I then went about trying to recreate this kind of dive where FBWA does not recover the plane. I have not been able to replicate it. I put the plane into manual mode, roll it over partially upside down and slap on FBWA and see if it can recover. One time it felt like it came close but still pulled out of it - so this is not resolved or explained, yet.
Well, that’s good news that it hasn’t repeated itself.
I’m just starting to look over the Ardupilot codebase, from a debugging point of view, whenever an operating mode is changed, there’s potential for a bug to come out in the code. I’m too green with the code to pin point the problem, but the mode change is a good place to start looking IMHO.
In order to attempt to reproduce, it would be good to set all the flight switches as they were, and fly at similar conditions (altitude, speed) and exclude sensors as probable causes.
I did not change the switch mappings on ardupilot’s end, only on my RC transmitter side just to be sure. Earlier I said I had, but I actually haven’t, apologies
I think I found out the cause of the crash:
Log File G:\00000051.log
Size (kb) 160622.396484375
No of lines 2027260
Duration 0:50:31
Vehicletype ArduPlane
Firmware Version V4.0.9
Firmware Hash a632c813
Hardware Type
Free Mem 0
Skipped Lines 0
Test: Autotune = NA -
Test: Brownout = GOOD -
Test: Compass = FAIL - WARN: Large compass offset params (X:-372.37, Y:97.65, Z:164.15)
WARN: Large compass offset in MAG data (X:-372.00, Y:97.00, Z:164.00)
Large change in mag_field (62.97%)
Max mag field length (770.34) > recommended (550.00)
Test: Dupe Log Data = GOOD -
Test: Empty = GOOD -
Test: Event/Failsafe = GOOD -
Test: GPS = GOOD -
Test: IMU Mismatch = GOOD - (Mismatch: 0.56, WARN: 0.75, FAIL: 1.50)
Test: Motor Balance = NA -
Test: NaNs = GOOD -
Test: OpticalFlow = FAIL - FAIL: no optical flow data
Test: Parameters = GOOD -
Test: PM = NA -
Test: Pitch/Roll = NA -
Test: Thrust = NA -
Test: VCC = UNKNOWN - No CURR log data
Is there any way to add failsafe measures so that this doesn’t happen again? There’s 2 IMU’s, and yet this one compass was able to down the craft both times when there was variance in it’s input.
I think that explains it. Very similar way of crashing, rolling over, pitch down and loss of control in FBWA mode. I rebuilt the plane, and today had a similar crash for a second time. The auto analysis in mission planner gives the same warning- too much compass variance (this time 42%). I think the wires I have running to the hatch with the compass in it must have moved, or something happened there and they got close to the compass. May have to replace the module just to be sure, and this time, move it into the wing far away from everything. How reliable are the modules themselves?
Compass issues is further confirmed as the cause of the crash by the documentation:
Accurately setting up the compass is critical because it is the primary source of heading information. Without an accurate heading the vehicle will not move in the correct direction in autopilot modes (i.e. AUTO, LOITER, PosHold, RTL, etc). This can lead to circling (aka “toiletbowling”) or fly-aways.
I’ve been using a Matek GPS & COMPASS MODULE M8Q-5883, are these known to have issues?
I suspect it is my installation layout to be the problem. I may disable the compass on the module, and purchase a separate compass module only just to be sure.
I use the M8Q-5883 modules in several planes and copters. They aren’t the best GPS/Compass modules out there, but they work well for me (for the price and size).
In plane you can disable the compass outright. It will work without it. The warning you shared is valid, but aimed at Copter. Once a plane is in motion the heading can be worked out by the GPS and EKF.
https://ardupilot.org/plane/docs/common-compassless.html#compass-less-operation