Plane 3.9.8 release


I have done the first beta release of the 3.9.8 stable release.

This is a minor release with some important safety fixes. The fixes relate to two issues:

  • implement hardware support to reset the CPU if a software or hardware failure causes the main loop to stop running

  • fixed a bug in the handling of a failure of the primary IMU in a multi-IMU system

The first fix relates to a flyaway that happened on a RadioLink mini-pix flight controller. The mini-pix suffered a major hardware failure that led to the main loop stopping. As the mini-pix does not have an IO co-processor this resulted in fixed control surface outputs and fixed motor output, which led to a flyaway. The plane was found,
but we want to ensure that if this ever happens again that the motor will stop and that the pilot will regain control of the aircraft.

The fix is to enable an option in the STM32 processor called “Independent Watchdog” (IWDG). The IWDG provides a mechanism to automatically reset the CPU on software or hardware failure resulting
in the main loop stopping. When this happens the hardware also provides a mechanism for ArduPilot to know that it is booting after a watchdog reset, in which case it does the following:

  • if the ChibiOS bootloader has been updated then it skips the normal 5 second delay in the bootloader

  • it skips baromoter, gyro and airspeed calibration, allowing for very fast boot

  • the home position and attitude estimate of the vehicle is restored to a point less than 0.3 seconds before the lockup

  • the pilot regains full control, and if the pilot requests arming then arming checks are automatically bypassed

We have tested this on an aircraft with a deliberately induced full CPU lockup. The aircraft recovered and flew normally within 3 seconds of lockup, with the pilot having full control. This was with a board with no IOMCU. A board that does have an IOMCU (such as a Hex cube) would have had full manual control on FMU lockup without the changes in this new release. For those boards with IOMCU the advantage of the new release is that the pilot will regain the ability to use stabilised and auto modes (including RTL) after a full CPU lockup.

The additional protections of the IWDG support only apply to the ChibiOS builds. The IWDG is not supported in NuttX builds.

The second key fix in this release relates to IMU failure on multi-IMU systems. If the IMU that is associated with the currently active EKF lane failed then the fixed wing attitude controller would lose attitude control and the aircraft will crash unless the pilot takes manual control. This has been fixed to ensure that IMU failover
operates correctly.

Other changes in this release:

  • added support for the CUAVv5Nano board

  • added retries to flash storage of parameters

  • fixed pullups on some fmuv3 based boards that lack hardware pullups on sdcard data lines

  • fixed fallback to microSD for parameter storage if a board with FRAM storage has a failed FRAM device

Happy flying!

7 Likes

Wow! Very interesting improvements. Thanks.

1 Like

thank you Tidge and all the developers,an other leap forward for Arduplane

Really love the IWDG system, thanks @Andrew_Tridgell!

1 Like

hi Marco, glad you like it. I don’t suppose you’d like to do a video showing a fixed wing aircraft continuing to fly after a complete lockup?

2 Likes

@tridge i‘d like to try :wink: how do i force a lockup?

Nice work @tridge! Wish my RC gear and RC brain weren’t so dusty, or I would volunteer to be a crash test dummy for IWDG.

Can someone tell me/us which of the popular ChibiOS boards (eg., F405-Wing) have an IOMCU, or not?

Kelly

Hi Basti,
in my modest experience, you best take this ESC on the F-405 Wing with Dshot enabled and wait for something like this.
:grin:

1 Like

@Rolf don‘t know if i‘m really confident enough for that kind of test… :wink:

@tridge found it. seems it’s skipping gyro cal on reboot, however FC stays in “initialize” after reboot unless kept still for a couple of seconds. how is that handled inflight?

what did you find?
You would need to use mavproxy and use the lock_autopilot command

the IOMCU boards have two banks of PWM outputs (eg. “main” and “aux”). For example, pixhawk, cube, CUAVv5, Pixhawk4, Pixhackv3 etc
All others have no IOMCU

thanks @tridge , i‘ll try that.

Is this intended to address the recent critical service bulletin for 2019 -manufactured Cube autopilots? Or is it unrelated, as the relevant pull request is still open?

yes, it includes a simpler fix for that issue (ie. the aircraft will not crash if it loses the active IMU)

1 Like

Mmmmmm… why not? :rofl:

2 Likes

well, apart from the possibility that it doesn’t work and your plane flies off into the sunset? that won’t happen of course :slight_smile:

Right now the only way to induce the lockup is to send a magic mavlink command from mavproxy. I could create you a release that adds a new RC option for locking up the autopilot, so you could put it on a switch. We decided against having that in the normal releases. Would that work for you? If so, what board do you want the build for?

Some notes:

  • if it is in AUTO it won’t restore the waypoint number its heading for, it would instead restart the mission
  • it won’t arm on reboot, but it will allow you to arm using the normal methods while flying (eg. GCS button or rudder+throttle)
  • if you are rolling/pitching rapidly when you reset then it will get confused for a few seconds (at least) and manual mode may be a good idea till it gets sorted out

The problem is the arming sequence if there’s a rudder installed.
The auto rearm could also be dangerous but more safe in that case, what do you think of using an rc channel for auto rearm after reset? Something like “if in a high state, over 1700 uS, autorearm the system after a reset”? Only for the testing phase, i’ve 6 free rc channels, form 9 to 14.
For the build “3DR Pixhawk” sounds good, initially I will do some bench tests.
About the risk, if the plane after the board reset work fine in full manual I think it’s minimal, the first tests would be better done in that way.
Everything depends on how many seconds after the reset the servos are back in operation.
For example an FX61 in Cruise at 70 meters AGL it could impact the ground after 6/7 seconds since the board reset.
If the procedure lasts longer it would be useless.

Thanks Marco for your amazing tests that most of Us don`t have the knowledge or courage to do but they contribute to fly safer :slight_smile:

1 Like

@Andrew_Tridgell, read the Skype chat…


I have now done the plane 3.9.8 final release. In addition to the changes listed above there were 3 more changes:

  • fixed the power brick2 status bit on FMUv5 boards
  • added a CubeBlack specific check for failed IMUs (to address the safety bulletin)
  • added logging to GPS receiver SD card in septentrio GPS modules

Happy flying!

3 Likes