Copter switches back to LAND mode from STABILIZE

Clem · June 6, 2019, 2:58am

My copter keeps switching back to LAND flight mode in mid flight or sometimes right after arming.
Here is a screenshot of the log with the mode changes (log is attached):

I thought this might be due to the “Err: FAILSAFE_EKF-1” error I’m getting in some of the logs but this EKF failsafe shouldn’t trigger a switch to LAND mode since my parameter “FS_EKF_ACTION” is set to 1 (LAND and not “Land even in stabilize”) which according to the documentation and the code (see below) shouldn’t trigger any action in non-GPS modes like stabilize and alt_hold.

// does this mode require position?
if (!copter.flightmode->requires_GPS() && (g.fs_ekf_action != FS_EKF_ACTION_LAND_EVEN_STABILIZE)) {
return;
}

Other things that could be triggering this:

GCS Failsafe is set to 1 and I have a companion computer sending proximity sensor mavlink messages to the Pixracer. Would a loss of communication with the companion computer trigger the GCS failsafe? … I don’t see any GCS failsafe message in the logs though.
all 3 of my compasses are disabled. This is on purpose as I’m using this setup for indoor flights where there is a lot of metal around the drone. Could this be triggering the EKF failsafe and then in turn the LAND?

My setup:
ArduCopter v3.7.0-dev running on Pixracer
Flight modes configured: stabilize and alt_hold (land is not part of the configured flight modes)
companion computer: Odroid XU4
all 3 compasses are disabled

Thoughts on what’s creating the problem?

2019-06-05 16-57-49.bin (488.3 KB)

xfacta · June 6, 2019, 3:39am

I see you’ve only got Stabilise and Alt_Hold set up, so it’s not an issue with RC ch5 putting it into Land mode (as you mentioned).

There’s a origin recorded and looks like good lat/long data so it should RTL if it goes into GCS failsafe - I’d expect it’d Land if it didnt have suitable position data - maybe that’s the issue: if it loses GCS comms (the companion computer) it also loses position data and cant RTL.

I can only cover those basics I think, so no real answers there - you’d have to wait for someone more experienced with indoor flight and EKF.

Clem · June 6, 2019, 9:13pm

Hey Shawn, thanks for taking a look at it.
Yeah let’s see if somebody has some insight on the possible role of the EKF here.

Clem · June 7, 2019, 4:37am

I’ve done more test flights with Mission Planner connected during the flight. Here is what I see:

During some flights the “FAILSAFE” message will display on the HUD even before takeoff but then I’m still able to take off, fly for a few minutes before the copter randomly switches back to LAND
Even with the 2 failsafes disabled (GCS and throttle) the copter will still randomly switch to LAND
Even after re-enabling the 3 compasses the random LAND issue persists
… not sure what’s going on.

peterbarker · June 7, 2019, 4:56am

Your arming checks are disabled. Why?

Clem · June 7, 2019, 5:28pm

Hey Peter, I’ve had a lot of test flights ruined because the copter wouldn’t pass arming checks and if I don’t have my laptop with me I can’t troubleshoot what the arming problem is so I’ve just disabled them. It’s probably not the best procedure
If that helps to troubleshoot i can reactivate them and see what happens.

Clem · June 7, 2019, 11:09pm

I just re-enabled some ARMING_CHECK parameters and did a few flights to test if enabling/disabling the compasses had any effect on the LAND issue. It turns out the random switch to land happens in all cases (with no compass, 1 compass or even 3 compasses active)…
On the last flight the copter landed but then I was unable to disarm it with the remote (copter kept switching back to land even after I switched to stabilize) so the props wouldn’t stop spinning… potentially unsafe
I had to do a force disarm from Mission Planner to get the props to stop spinning.
Screenshot of the flight:

Here is the log of that last flight

peterbarker · June 7, 2019, 11:39pm

I just re-enabled some ARMING_CHECK parameters and did a few flights to test if enabling/disabling the compasses had any effect on the LAND issue. It

Great! Would it help if we renamed this “SAFETY_CHECK” - because many of
these are. And they will often catch the sorts of things you are seeing
before takeoff.

turns out the random switch to land happens in all cases (with no compass, 1 compass or even 3 compasses active)…

Yes, your logs showed several problems:

On the last flight the copter landed but then I was unable to disarm it with the remote (copter kept switching back to land even after I switched to
stabilize) so the props wouldn’t stop spinning… potentially unsafe

That’s probably nothing to do with switching modes and everything to do
with the vehicle not knowing it is landed (won’t let you disarm unless you
are).

I had to do a force disarm from Mission Planner to get the props to stop spinning.

You might like to look at putting motor e-stop on a switch on your
transmitter.

I’ll have a quick look at your logs.

peterbarker · June 7, 2019, 11:53pm

One basic problem: massive magnetic interference on your mags (e.g):

            {
               "duration" : 301.29949951171875,
               "duration-units" : "seconds",
               "evidence" : [
                  "threshold=600.000000",
                  "threshold-duration=0.250000 seconds",
                  "length-max=1029.303223"
               ],
               "evilness" : 10,
               "evilness-is-deprecated" : "Use severity-score",
               "reason" : "Compass Vector Length above threshold",
               "series" : [ "MAG3.MagX", "MAG3.MagY", "MAG3.MagZ" ],
               "severity-score" : 10,
               "status" : "FAIL",
               "timestamp_start" : 107244903,
               "timestamp_stop" : 408544404
            },

That’s an unhappy graph. Poorly calibrated, for example!

Sorry, out of time right now to delve. But you probably need to move a mag out of the way of magnetic interference, and bless it as the one-true-mag.

You may also have balance issues (previous log mentioned pitch imbalance - but need a clean flight for that sort of thing.

You’re being changed into land as an EKF failsafe - the EKF is deeply unhappy with its outputs and Copter thinks the safest thing is to put the vehicle on the ground. This is absolutely exceptional behaviour - it is a very unhappy vehicle.

Peter

Clem · June 8, 2019, 2:26am

Thanks for taking the time to do the analysis Peter.
Yeah, SAFETY_CHECK might be a better term but don’t rely only on my opinion before making the change
Regarding the compasses, I’ve used the onboard calibration so it looks like that didn’t give good results (I used to use the live calibration before, will use that one next time). Puzzling thing is, even with 1 compass, I still have the LAND issue… I guess if the calibration is really off, the orientation calculated from the compass would barely ever change even in a 360 turn while the gyros would show a 360 turn… that would probably confuse the EKF.
We’re going to fly indoors inside of metal structures that’s why I had all compasses disabled at first. Do you know if the EKF can function with all compasses disabled?

Good observation on the pitch imbalance, we had to mount a bunch of sensors forward of the drone which tends to make the 2 front motors work harder. We’re planning on solving that on the next prototype.

Before you answered I had made 3 more test flights and observed the following: even with EKF failsafe disabled (FS_EKF_THRESHOLD=0) I still had a LAND event… This is odd because according to the code at line 38:

if (!motors->armed() || (g.fs_ekf_thresh <= 0.0f)) {
…
failsafe_ekf_off_event(); // clear failsafe
return;}

That should prevent the copter to act on any failsafe condition so maybe the LAND command is not coming from the EKF? For instance I have a log here that doesn’t show any failsafe message but the copter still lands.

After that I disabled GPS use by the EKF (EK2_GPS_TYPE=3) and I had 2 flights without LAND event… not sure if there is a causal effect or if that’s just random but that only deepens the puzzle

Thoughts?

peterbarker · June 8, 2019, 11:27pm

A little more time to look at this this-morning.

2019-06-08 05:07:35.77: MODE {TimeUS : 106818429, Mode : 0, ModeNum : 0, Rsn : 1}
2019-06-08 05:07:36.11: MODE {TimeUS : 107159018, Mode : Stabilize, ModeNum : 0, Rsn : 1}
2019-06-08 05:07:42.87: MODE {TimeUS : 113924267, Mode : AltHold, ModeNum : 2, Rsn : 1}
2019-06-08 05:11:41.70: MODE {TimeUS : 352747154, Mode : Land, ModeNum : 9, Rsn : 2}
2019-06-08 05:11:46.98: MODE {TimeUS : 358034331, Mode : Stabilize, ModeNum : 0, Rsn : 1}
2019-06-08 05:11:48.71: MODE {TimeUS : 359754785, Mode : Land, ModeNum : 9, Rsn : 2}
2019-06-08 05:12:09.19: MODE {TimeUS : 380244136, Mode : AltHold, ModeNum : 2, Rsn : 1}
2019-06-08 05:12:09.73: MODE {TimeUS : 380784076, Mode : Stabilize, ModeNum : 0, Rsn : 1}
2019-06-08 05:12:10.73: MODE {TimeUS : 381774788, Mode : Land, ModeNum : 9, Rsn : 2}
2019-06-08 05:12:20.42: MODE {TimeUS : 391474048, Mode : AltHold, ModeNum : 2, Rsn : 1}
2019-06-08 05:12:20.79: MODE {TimeUS : 391844284, Mode : Stabilize, ModeNum : 0, Rsn : 1}
2019-06-08 05:12:21.74: MODE {TimeUS : 392787167, Mode : Land, ModeNum : 9, Rsn : 2}
2019-06-08 05:12:24.24: MODE {TimeUS : 395294048, Mode : AltHold, ModeNum : 2, Rsn : 1}
2019-06-08 05:12:24.71: MODE {TimeUS : 395764106, Mode : Stabilize, ModeNum : 0, Rsn : 1}
2019-06-08 05:12:25.09: MODE {TimeUS : 396144259, Mode : AltHold, ModeNum : 2, Rsn : 1}
2019-06-08 05:12:25.74: MODE {TimeUS : 396794320, Mode : Land, ModeNum : 9, Rsn : 2}
2019-06-08 05:12:25.92: MODE {TimeUS : 396974042, Mode : Stabilize, ModeNum : 0, Rsn : 1}
2019-06-08 05:12:26.74: MODE {TimeUS : 397794125, Mode : Land, ModeNum : 9, Rsn : 2}
2019-06-08 05:13:14.92: MODE {TimeUS : 445974093, Mode : AltHold, ModeNum : 2, Rsn : 1}
2019-06-08 05:13:15.40: MODE {TimeUS : 446454124, Mode : Stabilize, ModeNum : 0, Rsn : 1}
2019-06-08 05:13:15.79: MODE {TimeUS : 446838716, Mode : Land, ModeNum : 9, Rsn : 2}
2019-06-08 05:13:17.31: MODE {TimeUS : 448364612, Mode : AltHold, ModeNum : 2, Rsn : 1}
2019-06-08 05:13:17.57: MODE {TimeUS : 448624312, Mode : Stabilize, ModeNum : 0, Rsn : 1}
2019-06-08 05:13:17.79: MODE {TimeUS : 448838703, Mode : Land, ModeNum : 9, Rsn : 2}
2019-06-08 05:13:18.21: MODE {TimeUS : 449264016, Mode : AltHold, ModeNum : 2, Rsn : 1}
2019-06-08 05:13:18.57: MODE {TimeUS : 449624124, Mode : Stabilize, ModeNum : 0, Rsn : 1}
2019-06-08 05:13:18.79: MODE {TimeUS : 449844668, Mode : Land, ModeNum : 9, Rsn : 2}

So… that’s interesting. It’s changing into Land because… the GCS is telling it to. Does that make any sense to you? The .tlog may be illustrative / provide confirmation.

Clem · June 10, 2019, 2:06am

Outstanding! That explains everything. I had a piece of code buried in my companion computer’s code that automatically lands the copter under a certain battery level. That explains why the LAND action was triggered even with the EKF failsafe disabled.
I’m going to do more flight tests before I validate the fix but thanks a ton for the detailed analysis Peter!
I didn’t know the MODE log message had a “reason” field in it. The documentation didn’t mention it so I’ve changed it and created a PR. https://github.com/ArduPilot/ardupilot_wiki/pull/1784
Cheers

Clem · June 12, 2019, 4:39am

I’ve deactivated the LAND function on my companion computer script but the LAND behavior has kept happening… then I noticed the following error message in Mavproxy on my Odroid: " Exception in message handler for HEARTBEAT mode 0" which led me to that post: https://github.com/dronekit/dronekit-python/issues/677
Since then I’ve flown 2 flights without having Mission Planner connected and the LAND issue seems to have disappeared… I’ll need to do more flights to confirm but would it make sense that messages coming from Mission Planner were triggering the LAND behavior?

Clem · June 17, 2019, 10:06pm

I’ve done more test flights without connecting the GCS (ground laptop) and the land issue has totally disappeared so it’s possible there was some crosstalk between the mavlink packets sent by the laptop and the ones sent by the onboard companion computer.
That seems to have solved the problem!