AC 3.6.9 Crash with IR-Lock and PLND_EST_TYPE enabled

mtbsteve · July 16, 2019, 9:32pm

After 2 years this defect was initially reported, it is still open with Arducopter 3.6.9 along with the IR-Lock sensor for precision landing.
When PLND_EST_TYPE is set to 1, the copter gets out of control and goes into a uncontrollable spiral when Precision Loiter is enabled once the IR beacon gets in sight. Unlike during Land mode where you can abort the spiraling by switching into another flight mode, disabling precision loiter does not bring back control.
As long as this defect has not been fixed, please do not enable the kalman filter and keep PLND_EST_TYPE set to 0.

Link to the issue and latest log:

Anubis · July 16, 2019, 11:20pm

Can you expand on this part a bit? Do you mean switching from PLoiter to Loiter doesn’t stop the death spiral? Stabilize/alt hold?

And as for the issue, as I understand, truly fixing it requires a significant refactor of the precision tracking code, which is why it’s taking so long. @fnoop understands the scope and has done a lot of work on it already, I believe.

That said, I agree that in the meantime the PL EKF option should be hidden or at least have some warning that it is experimental and possibly dangerous.

fnoop · July 17, 2019, 7:22am

I did the original refactor nearly 2 years ago:

github.com/ArduPilot/ardupilot

AC_PrecLand: Add buffer parameter and correlate timestamps, rebased, …

ArduPilot:master ← fnoop:acprecland_buffer

opened 04:54PM - 07 Oct 17 UTC

fnoop

+8215 -2583

…fixes #6297 #5959 This adds a 'PLND_BUFFER' parameter that can be used to se…t the length of an inertial buffer within AC_PrecLand. This was previously fixed at 20ms and fixed to use the last available inertial frame, so effectively locked the latency between the two systems (Flight Controller and Measurement Sensor) to 20ms, and appears to have been set to work with the IRLock system. However, most visual systems using the companion backend will typically have latencies more in the 50-200ms range and so the attitude compensation is applied to long-lost inertial frames and causes dangerous behaviour. The PLND_BUFFER now influences the length of this buffer and is set to 20ms by default, so will behave in the same way as before, but it can be setup up to 250ms (and the buffer is changed immediately if the parameter is changed), to allow slower/higher latency systems. In addition, this commit also now honours the 'time_usec' field of LANDING_TARGET mavlink message, and extends AC_PrecLand to consume this timestamp in a way that other backends can also be extended to use (_backend->get_los_timestamp()). If the timestamp is set, it attempts to correlate the closest inertial frame to the sensor frame. If it cannot correlate the frames (if the frame is out of bounds of the inertial buffer), the measurement is ignored. If the timestamp is not set, it continues the previous behaviour of blindly using the oldest inertial frame. In conjunction with the earlier TIMESYNC work, this should now allow AC_PrecLand to automatically adjust to latencies dynamically, if the backend provides a synchronised measurement timestamp. vision_landing (https://github.com/fnoop/vision_landing) is such a system. This is not a perfect finished system yet, by any means. There are still numerous issues to address, but this gets us one (or two) steps closer. A few notes: - Changes inertial buffer from the fixed-at-compile-time AP_Buffer to AP_HAL RingBuffer which can be changed on the fly, uses recent PR #6963 - Debug output is present, will remove in a subsequent commit - TIMESYNC/MAVlink mechanism is very sub-optimal, huge jitter and latency inherent. Needs improving (PR to come) - The sending sensor (companion computer/camera) is still responsible for capturing a synchronised timestamp at the time of frame capture

I did another deep dive a year ago:

Most of the PRs are still sitting waiting to be reviewed.

I don’t have an irlock so don’t know if the problems with irlock are related, however @rmackay9 made this comment on one of the PRs:

As a side note, this PLND_LAG parameter makes it easy to reproduce the bad behaviour users see when the lag does not match their sensor’s lag. If we set PLND_LAG = 160 in SITL we get absolutely crazy roll and pitch wobbles. It’s horrifying actually.

If you have a particular problem with irlock, being a commercial product, can they not help you?

mtbsteve · July 17, 2019, 9:22am

@Anubis I can confirm that switching from PLoiter to Loiter does not stop spiraling. I have not tried to switch to stabilize simply because I wasn’t fast enough until the copter crashed to the ground.

mtbsteve · July 17, 2019, 9:36am

Thanks for your support!
@ThomasSFL - the founder of IR-Lock - has always been extremely helpful. However here I assume a software defect in the PL code.in Arducopter rather than a hardware probem. With PLND_EST_TYPE set to zero IR-Lock works well, however when you enable the kalman filter, the behavior of the drone can become critical, and especially in PLoiter you cant escape the death spiral even when you switch back to loiter.
So as long this is not being fixed, I suggest to add at least a warning note on the usage of the PLND_EST_TYPE parameter to the Precision Landing wiki.

ThomasSFL · July 17, 2019, 1:09pm

@mtbsteve @fnoop

Thank you for your feedback.

My experience has been that PLND_EST_TYPE = 0 has always been the most robust setting for a wide range of copter builds using IR-LOCK. We recommend PLND_EST_TYPE = 0 on our wiki.

The EKF control method (=1) has worked great for some copters, but it also has a chance of producing unstable control. The EKF PL method (=1) was actually developed for a particular commercial build. It worked great for them, but I don’t recommend it as the default setting.

Best,
Thomas

fnoop · July 18, 2019, 11:21pm

I agree, also a warning for any vision based system. I have a big fat warning on my vision_landing repo.