170623 – DJI S1000 frame
Heavy lift capable UAS.
DJI S1000 frame / Pixhawk 2.1 Flight controller / Mission planner beta GCS / Firmware 3.5.0.rc7
The incident happened on the 8th test flight of the day after completing seven successful test flights to adjust throttle settings, yaw settings, compass settings in Alt hold, and Loiter mode mode because of an aggressive throttle on take off. Before mishap aircraft was responding normally other than the occasional toilet bowl in Loiter mode.
Preflight arm checks passed, no warnings on GCS screen. Intermittent Gyro Health warning that was self-cleared before arming. Safety switch was turned off, aircraft was armed in Loiter mode which is GPS guided. We had what is referred to as toilet bowling after take off which is a compass variance error in the flight controller. The aircraft was place in Althold mode which disables the GPS and solely relies on the barometer for altitude control. The aircraft responded normally to Althold for approximately three seconds and then aggressively accelerated and departed to the SE at 34 mph un-commanded until ground impact.
At last effort I asked the GCS operator to go to Stabilize mode for pre discussed fly away procedure, but aircraft hit the ground before command could be executed. Also at last ditch effort the aircraft was placed in Return to Launch mode right before impact. Fly away in ALTHOLD MODE?
All commands, flight modes, and observations were validated in the flight controller telemetry log.
can someone review the logs to tell me if they can see anything?
On a first look, at aprox 3 seconds after ALTHOLD a battery failsafe kicked-in. Will have to look a bit further.
Also taking off with a depleted battery (aprox 23V) on a S1000 is risky, since it consumes around 60A and goes easily to 80A. I land our S1000+ at 23.5V - I also used to fly with 22000 but changed to 16000 with better results
Thanks for the report and sorry about the crash. As @LuisVale says, the vehicle switched into RTL but it was not able to control it’s position so it flew off in the wrong direction. It gets up to a whopping 16m/s just before the crash.
Both EKF’s were in heading fusion mode at the time. In this mode magnetometer errors are harder to detect because the 3-axis field measurement is projected into the earth frame and reduced to a single magnetic compass yaw measurement.
The copter hadn’t climbed past the trigger height of 5m which causes it to transition to 3-axis fusion.
As a result the compass innovation test levels never climbed past 1 during the event and therefore the check condition in line 116 of failsafe.cpp was not achieved.
There was however evidence of a very large level of magnetic interference affecting Compass 1 that caused the mag yaw innovation check to fail during initial climb and reset the yaw and field states. given the amount of magnetic interference, I am surprised it flew in loiter as long as it did:
The criteria used in failsafe.cpp needs to be revised because we cannot rely on magnetometer innovations for low altitude operation before the switch to 3-axis fusion, particularly if there is interference rotating the field in earth frame.
Recommendations
Check the wiki for first flight instructions to make sure we reinforce the need to do a ground run and check for compass interference.
Change the switch to 3-axis fusion in the EKF from 5m to 2.5m
Rework the criteria in failsafe.cpp. to remove the need for a failed compass innovation if the GPS position and velocity test ratio is high. eg
After discussing with Paul, we’ve raised a PR which should help should this happen in the future.
reduce the altitude of the EKF’s final compass reset from 5m to 2.5m. Above this altitude, all 3-axis of the compass are used when calculating the compass variance (below this altitude only a projection onto the earth-frame horizontal plane is used). With all 3-axis used, a bad compass will be detected more quickly.
trigger Copter’s ekf variance when any two of three variances (position, velocity and compass) are over the threshold. Previously only velocity and compass were used.
@telestroud, thanks again for the report and sorry for the troubles. If it’s any consolation, hopefully this crash will actually help us close these corner cases and improve overall safety.
This PR is in master now and we’ve included the first part of the fix, “reduce the EKF’s final compass reset from 5m to 2.5m” in AC3.5-rc10 which should appear in the mission planner for beta testing within a few hours.
I didn’t include the ekf variance check because we need more testing to ensure we don’t get too many false positives and we are too close to the AC3.5 release. We could hold up the AC3.5 release but we think this incident could have occurred in AC3.4.6 as well. So we just need to march forward with the AC3.5 release and then get the improve ekf check out with AC3.5.1 as soon as it appears safe.
- I also used to fly with 22000 but changed to 16000 with better results 
