Proposal: Distributed control loops with virtual/actual peripheral communication or MAVLink

General Rants&Raves
1

[EDIT] I was not aware that porting Ardupilot to ChibiOS and HAL_ChibiOS had already been completed last year when I wrote the first post, so most of the initial discussion is irrelevant now. Go directly to the #10 post for a revised proposal which take into account the fact that Ardupilot is already based on a portable OS with hardware abstraction.

[EDIT bis] I don’t think this thread really belongs in the “Rants&Raves” forum but there is no “Architecture” or “Long-Term Evolution” forum. If a moderator think that this thread belongs in another forum, feel free to move it, I only used the “Rants&Raves” forum because I didn’t want to pollute a forum dedicated to other topics.

Hi,

It’s neither a rant or a rave but I didn’t want to pollute specific forums with this subject:

Ardupilot has, so far, been a single MPU architecture, requiring a relatively high-end MPU for the fast control loops (make your drone stable) while handling at the same time the slow loops (remote control input, OSD, route planning, etc.). This approach makes Ardupilot hardware more expensive than necessary considering its limitations (no computer vision, or AI flight control optimization, or anything you could do on a linux/FreeRTOS platform).

I was planning on developing an F3/Orange Pi zero flight controller (same price or lower than an Pix4) with Lora transceiver to solve the receiver/vtx interference problem and provide variable video bitrate based on RSSI, and allowing more capabilities for the low rate control loops in the future, like follow-me mode, subject-centered pan, OpenCV based optical flow control for the gimbal/drone, etc. I know its feasible with a Pix4 plus and something-Pi but it costs twice as much as necessary, and require extensive modification of the Ardupilot code to delegate all the slow control loops to a separate computer.

Of course, I could develop it all by myself, it would only take 3 years to become proficient enough in C/C++ (I’m more a XXIst century language guy, pyhton, go, rust), plus an additional 2-3 years to learn everything about the Ardupilot architecture and port/debug it on a MPU/Pi platform. Or, I can present a proposal and people already proficient in C/C++, who know the architecture of Ardupilot, could do it in months rather than years (Yes, years, because I also have a job and other hobbies).

So, here’s my proposal for the next generation Ardupilot hardware architecture:

  • The cheapest hardware to stabilize the drone in a fast (100-500Hz) control loop (only accelerometer/gyro, with an attitude feedback to a gimbal controller, something a cheap 6DoF F3 controller can do easily);
  • A Pi zero or anything which runs FreeRTOS for the slow (5-10Hz) control loops (altitude stabilization, GPS, image flow control, IP transceiver, planned route, GCS, follow-me mode or any computer image vision mode, AI based auto-tune, etc.)

The main advantage would be the functionality/cost ratio (a basic 6DoF F3 controller plus an Orange Pi zero cost less than a Pixhawk), easier developement (mixing fast 100-500Hz control loops with complex calculus 5-10Hz control loops is much easier on two MPU/CPUs linked with SPI than on a single one), and much more capabilities (a 5Hz control loop allows IP transceivers like Lora for control and video on the same frequency, basic OpenCV/AI image recognition or flow control, etc.).

So, in a nutshell, using a cheap MPU for stabilization and a cheap linux/FreeRTOS for the slow control loop would open Ardupilot to a vast array of new capabilities like professional video , high speed/resolution stereoscopy obstacle avoidance, easy IP-based long-range setup without RTX/VTX interference, etc., all for the same hardware cost as a current Pix4.

I can contribute to the architecture if necessary but, first, I think we should discuss the advantages/drawbacks of a two-platforms architecture instead of a single architecture for the future of ardupilot.

P.S.: By the way, that’s how we, and most mammals, have evolved. We have specialized brains for every tasks, so each brain is optimized for each task, and they communicate (at a very low bit rate) to coordinate their own tasks. It starts to be applied to robots too.

2

Hello,

It seems that there a some misconception about ArduPilot and compatible hardware, and hardware capability in general.

Linux SoC aren’t that good for realtime task, specially when you try to mix realtime constraint and vision stuff. Moreover, even if some ARM SoC are cheap, they aren’t reliable when microcontroller are.

About ArduPilot, we already support linux flightcontroler for long time. Look on the wiki to find some linux board that already got the architect that you are speaking about.

If you are interested in distributed architecture, ROS will be a more interesting architecture IMOH. But the strong point of ArduPilot is that it does it tasks nicely and hide most complexity.

You could also look at CAN support that will allow to distribute task around multiple controler (ideally)… When we found more people and industrial to develop it. Otherwise there is also AP_peripherical that will does a part of the job.

Don’t trust the big number in front of CPU or new technology like AI… ArduPilot works on MCU because it is quite well done and don’t need a Nvida chip to work…

About board price … It depends the quality you want. Some hight end controler aren’t that cheap, but are truly reliable and got real company behind them. That comes with a price but for good.
You can also find 20$ controler or made one on allwinner cheap for example…but good luck to have them reliable!

That was just my thoughts, not the Truth. :stuck_out_tongue_winking_eye: I am pretty some will defend linux flightcontroler :stuck_out_tongue_closed_eyes:

3

There are several low cost boards that run Ardupilot just fine. Many are using ~$25 F4 boards in multirotors and planes. $50 gets you an F7 board. There are inexpensive boards with 2 IMU’s.

4

Hi,

I agree with you that Linux isn’t the best for fast control loops and, even on a real time OS like FreeRTOS or ROS, mixing fast control loops and complex tasks reliably is not easy. That’s the reason why I was suggesting an MCU+SoC architecture, with fast control loops on the MCU and slow control loops/complex tasks on a SoC, rather than porting all of Ardupilot on an SoC with real-time OS.

In short, what I was suggesting was to push further the concept of “companion computer” already implemented in Ardupilot: instead of using the “companion computer” only for computer vision or similar complex tasks, the companion computer would take care of the slow control loops. This would lower the requirements for the MCU who would only take care of the fast control loops and provide ample computing power reserve - especially if the SoC GPU supports OpenCL - for the future evolution of Ardupilot. The example of a basic F3 flight controller plus an Orange Pi zero was just to show that a MCU+SoC architecture would not necessarily increase the cost of hardware (except for the weight cost). The user could always select a more powerful or proven SoC platform if required by his application.

As for reliability, there is no inherent difference in reliability between an MCU board and a SoC board. It’s the OS and/or the implementation of the SoC that introduce reliability problems. But a good implementation of an SoC and a good implementation of an OS-based software can be as reliable as a MCU firmware. Ardupilot is not reliable because it runs on an MCU, it is because a lot of people spend a lot of time to make sure there was no bug nor conflicts in Ardupilot. The same can be done for an OS based software, just look at all the internet servers you rely on daily (or the 21 virtual servers that have been running 24/7 for 3+ years at my place - yes, I know, it’s way overkill for a NAS but I can somewhat justify them if I really really try hard ;-).

Beside allowing long-term evolution of Ardupilot, a MCU+SoC architecture would easier to port to new hardware platforms since the OS provides an abstraction layer for the part of Ardupilot that runs on the SoC. Only the fast control loops would need changes in the code to support new MCU platforms while, in most cases, the SoC part would only need new compiling options but no changes in the code.

Now, I also recognize the advantages of a single MCU architecture for simple applications (lower weight, easier implementation) but if all you need is a flight controller with RTH, we are already covered by the like of LibrePilot or Betaflight. It is my understanding that Ardupilot targets more complex applications, like mapping, aerial delivery, disaster recovery, etc. These applications would benefit from native object recognition in Ardupilot for example, something that would not be possible on an MCU platform (except with the like of i.MX6 but that’s another level of cost).

As I said, the “companion computer” already allow tasks too complex for a STM32F4 to be executed on a more powerful platform and interact with Ardupilot via MAVLink but this approach doesn’t solve the problem that Ardupilot will require more and more powerful MCU platform when it evolves in the future. It also means the user is on his own if he wants more than basic flight control and way points. Complete mapping/delivery/disaster recovery solutions should be in the future of Ardupilot, in my opinion, although we are talking long-term future in that case. But planning for and allowing this future should be a current concern.

As you said yourself, this is just my opinion, other people can - and will - disagree with my point of view. An MCU+SoC architecture is not the ultimate panacea for flight controllers (it weights more, takes more space, and require two different expertise to develop) but I think the advantages would outweigh the drawbacks for the specific target Ardupilot is aiming for. For simple flight control and way points, there are other open source solutions today. In my opinion, Ardupilot should remain the more “serious” alternative, able to handle more complex tasks, like delivery to a visual target, locating poachers, or victims in a disaster, for a cost even less privileged countries can afford.

P.S.: DMIPS and GFLOPS are not big marketing numbers you can’t trust. They don’t say everything, but they are still a good indication of the computing power.

P.S. bis: AI (or machine learning if you prefer) is not the panacea the current hype sells but it does excel at certain tasks, especially solving problems which are hard to solve with a classic algorithm. Object recognition is one of the best known example, but machine learning also give better results than known algorithms for problems like auto-tuning PIDs in a complex system. With 18 DoF (3 rotation positions, speeds and accelerations and 3 lateral positions, speed and accelerations), 10 DoF feedback loops and 18 DoF perturbations (wind), a drone is a complex system. If it was easily solvable by algorithms, all drone would be as stable as DJI’s.

5

Hello,

Have you looked at existing Linux systems that are already running Ardupilot ?

  • RaspBerryPI : NAVIO
  • BeagleBone : BBBMINI

These can process basic companion computer stuff and Flight Controller functions on the same platform. After more than 3 years of active support on these systems I can personally confirm that this is way more difficult to make it work than the FC+CC combo.

The challenge to master ArduPilot and Linux with the ever problematic Near Real Time Kernels makes it a steep learning curve. Add to that, the non existent support because so few of us have wandered into uncharted waters , makes it even harder to implement.

So I consider the actual architecture as optimal because you deal with a stable FC and you can experiment with different types and complexity of missions using a balanced CC, ranging from a Pi Zero to a NUC I7 and associated peripherals , like what we are doing with the T265 on the GSoC project this summer.

6

Hi,

To be frank, I have ignored solutions solely based on Linux because using a non-real-time OS for real-time control loops is asking for trouble, especially fast control loops like gyro stabilization. That’s the reason why I suggested to keep the gyro stabilization loop on a MCU so the control loop frequency won’t be affected by unpredictable interruptions, then use a real-time OS (FreeRTOS or, as khancyr suggested, ROS) for the slower control loops who reside on an SoC so even those have a predictable update frequency.

The only problem I foresee is the interaction between the CPU and GPU (for variable bitrate video), and possibly a cryptoprocessor (for encrypted control and feedback when needed) since these “coprocessors” rely on interruptions for communication with the CPU. However, since the CPU load of an SoC would be minimal compared to an F4 MCU, I think handling GPU and cryptoprocessor interruptions within the slow control loop cycle wouldn’t be too much of a problem if the SoC only handle slow (less than 100Hz) control loops.

As for the uncharted territories, I am fully aware of it.That’s why I proposed this architecture to a group of people who are way more knowledgeable than me as a group rather than trying to tackle the challenge myself.

On the other hand, using the current FC+CC architecture would also be an option, but to achieve the “controlled cost” goal, it would mean any future evolution of Ardupilot that requires more computing power would be developed on the companion computer in order to avoid the pitfall of always increasing the MCU requirements.

In the current state of technologies, gyro stabilization (i.e. relative angle position, which is equivalent to angle speed) on a cheap STM32F3 and all the integrated control loops (i.e. angle absolute position - accelerometer - which is related to lateral acceleration, plus all the integration from lateral acceleration to lateral position) moved to an SoC, seem to provide the best cost/benefit ratio in term of hardware.

So the main question at the end is: Is the effort of splitting the current Ardupilot control loops between an F3 and an SoC running an R.T.OS worth the effort, or should Ardupilot keep all the control loops on an F4 MCU and develop everything in the future on a CC SoC?

I tend to favor low-cost hardware and high-cost development because you develop only once but 10s or 100s of thousands of people will have to live with the hardware cost. But that’s just my opinion.

7

I think you still mix many technology.
Ardupilot linux HAL is using realtime kernel. That soft realtime but that as good a freertos (that isn’t an OS but a realtime library, you are in charge of dealing with the OS). It can easily handle some sensors at fast rate.
Parrot or DJI also do that… They got linux flight controler with realtime kernel and it works fine.
You even got some SOC with dedicated realtime hardware accelerator ( see BeagleBone Black )

On other hand, ROS isn’t an OS nor get any piece of realtime features.

My vision on ArduPilot is that we should have a trusted flight controler. So that is better to restrain its capacity on what is working and well tested. All other stuff that aren’t mandatory belong to companion computer.
You can always have the CC to send direct motor control to the flight controler. But is case of issues ( that will come fast with CPU ,AI, etc) the flight controler is always able to finish the mission. That a true advantage ! That something already largely done. CC isn’t only for logging and visual stuff.

The cost factor isn’t an issue as even H7 controler are cheap… But Soc have a much higher learning curve and complexity.
And issue on SoC isn’t only due to OS… Check better… Allwinner soc are crappy ( and support is inexistent) even raspberry pi is throttling at hight load.
ArduPilot is an autopilot for small autonomous vehicles, so unfortunately we cannot deal with OS matter. (And we really lack of workforce)

In short, we already support what you are speaking of,but we still try to push even better the capability on the MCU as the code complexity is simplier. If you look on ArduPilot, we still get a bunch of CPU time available on F4 controler, and are being able to put even more capability through good refactoring. ( I think that between copter 3.6 and 3.7, the space and ram consumption don’t raise…)

8

Hi,

I agree with your point about an independent FC able to bring your drone home (or any other failsafe behavior) rather than just the gyro stabilization which is only useful if your drone is line-of-sight. But it’s also another argument for a distributed architecture: If your basic FC is “critical”, i.e. you rely on it to retrieve your expensive drone, you don’t want every new feature, so possibly bugs, to run on the same CPU as the critical part of the FC.

As for the complexity, not all OSes are as feature rich, and therefore not-that-reliable and complex to master as desktop/mobile OSes. An OS designed for control, like FreeRTOS or mBed, is barely more complex to use than Arduino. And, yes, FreeRTOS is an OS: it has its own preemptive kernel, its own HAL and runs on bare metal without relying on another OS or kernel. The fact that the resulting binary is monolithic (kernel + HAL + application) rather than modular like typical desktop or server OSes doesn’t mean it’s a library.

Also, if the real-time OS is limited to the basic features you typically need for real-time applications, its code is small (A basic FreeRTOS runs with room to spare on a Cortex M0) so easy to debug. Since the same OS is used for a lot of various applications, it has been tested and debugged by much more people than dedicated applications. And I think an FC is not really as critical as a medical equipment, so if the FreeRTOS is certified for medical applications (under the name SafeRTOS), it should be reliable enough for an FC.

Since the kernel and hardware abstraction layer has already been ported on several platforms (FreeRTOS runs on basically any Cortex from M0 to A9 - MCUs and SoCs - plus a few non-ARM MCUs and CPUs), the time you spend to port the code on the OS will be more than saved in the future because of simpler porting, usually by only changing the compiling options and the IO pin assignments (once again, unlike desktop/mobile/server OSes where porting is always like opening a Pandora box).

Of course, porting legacy bare-metal code to an RTOS takes time, that’s why the title of my post says “next generation”, not “next version”. Changing from a monolithic software architecture to independent, portable modules linked by MAVLink may involve more work. It may add to the effort required during the next refactoring but I think it would be worth it:

People who only need basic features like waypoint navigation could still run the entire FC on a single MCU, using RAM-queues for the MAVLink pipelines, but people who want to develop more advanced features could contribute them to the Ardupilot ecosystem even if they are too complex to run on a 32 bits MCU. And you wouldn’t have the headache of keeping the basic FC reliable on a non-preemptive OS while people keep adding more and more complex feature.

It’s not that a complete FC on Linux (or any other fully-featured ,non-preemptive OS) is not feasible, but I don’t think the Ardupilot project has the same full-time resources, and especially the quality control team, of a DJI or a Parrot. Maintaining the reliability of your critical control loops on a non-preemptive OS while complex task like subject tracking run on the same OS and CPU is not exactly easy to develop or debug. A MCU with your time-critical loops, linked to another platform/OS with a message queue, is a more robust solution: if someone screw up when developing/improving a complex feature, your drone remains controllable because the time-critical control loops won’t be affected. It’s also easier to debut because you can run the new feature on a separate board (no conflict) until it’s reliable enough to move it to the main or second board.

The single CPU/MCU/SoC is the go-to architecture for most people simply because that’s what we see everyday. Until recently, distributed architectures were only considered when the task was too complex for a single computer and you didn’t have the cash for a supercomputer. But a distributed architecture is also a robust and extensible solution for robots (a drone is just a robot that flies after all), and makes real-time control plus complex tasks easier to implement since your time-critical control loops can be entirely independent from the rest of the solution.

Maybe just to keep in the back of your mind until the time for the next major refactoring comes. As I said, “next generation”, not “next version”.

Ardupilot was once the best, most feature rich FC, but fell behind most closed-source solutions since. If adding complex features without breaking the basic FC was easier, features like the 120fps obstacle avoidance with dynamic route planning that MIT students developed 4 years ago could make their way into the Ardupilot ecosystem instead of being slowly forgotten on a neglected Github project (that’s just one example of features already developed which could bring Ardupilot back on the top), and I could also contribute the control/telemetry/video on a single transceiver, with variable bit rate for extended range, when/if it works.

P.S.: For ROS, my bad, I implicitly think of ROS as ROS+RTOS but I should have been more explicit.

P.S. bis: MCUs are CPUs plus additional features, same as SoCs except the additional feature of MCUs are more focused on control, the SoCs are more focused on desktop/server use. But, at the end, it’s only the feature set that’s complex, and only if you want to use all of it. That’s why fully-featured OSes like Linux are so complex and therefore not always reliable, but it has nothing to do with the CPU. Also, throttling is not a bug, it’s a feature in case you have undersized your cooling and is preferable to the alternative: burning your CPU and the resulting drone crash that will follow. Most mobile platforms throttle down for that reason: people like big numbers but don’t want to carry the volume of proper cooling for a CPU at full load. But it’s fairly easy to circumvent in a RTOS: select another clock than the CPU clock for your ticks, and make sure your time-critical loops are prioritized. When the CPU gets too hot, your drone may take a bit more time to correct slow derives, or your compressed video may have more latency, but the drone will still be as stable as usual.

9

I just noticed that porting Ardupilot on an RTOS was just completed last year, using ChibliOS rather than FreeRTOS. ChibliOS is a fine RTOS too, supporting a decent range of platforms including simulators on mahor OSes, and apparently a more consiistent HAL across platforms. So the RTOS part is done.

What remains is splitting Ardupilot into a basic FC plus extended feature modules, which would also be useful for the H7 platform if the target H7 is the dual core M4+M7.

10

I have changed the title because the discussion we had (thanks) have changed my perspective on my initial proposal, more specifically:

  • The RTOS discussion is not relevant anymore since Ardupilot has already been ported on a RTOS (ChibliOS) which solve the portability aspect of the proposal. I missed that part when I looked at the roadmap, my bad. ChibliOS doesn’t have much support for more powerful platform than typical MCU platforms but porting it to “linux boards” or equivalent doesn’t seem that bad, especially considering the new ChibliOS HAL is based on HAL_Linux.
  • With ChibliOS comes DMA peripheral drivers, so virtual peripherals (as in simulated in RAM, not as in virtualization) could be an efficient way to link independent control loops since virtual peripherals in RAM can be almost as efficient as shared memory blocks. MAVLink is also an option but I don’t the overhead and latency that MAVLink introduces compared to virtual peripherals.
  • Khancyr helped me realized that gyro stabilization (i.e. being able to bring your drone back home manually) may not be sufficient for pilots with “advanced” license who are allowed to fly outside the LOS limit. Beside, presentations from the last video conference made me realized that Ardupilot is not only used for “serious” use case, it has also become a solution for low-cost consumer grade drones. So being able to still run Ardupilot on a low-cost, single MCU platform is not something to neglect.

So here’s the revised proposal:

Define a critical core for time-critical loops necessary to bring your drone home. The criticality could have two level:

  • basic gyro stabilization allowing you to bring a drone back home manually. That core would always reside on the main MCU because jitter in gyro stabilization can really mess up the ability to control the drone, especially on multi-copters. This core needs the highest efficiency and needs to be as independant as possible from new features, so it would use low-level RTOS messaging or shared memory whenever necessary, at the expense of portability to non-preemptive OS.

  • RTH core, which would normally reside on the main MCU but, since speed and position are not as time-critical as gyro stabilisation, this core would use virtual/actual peripherals (or MAVLink) so control loops in this secondary core could be distributed if you want to, by simply changing the peripheral/hardware definition table (or the hardware pipeline for MAVLink, if MAVLink is used).

Last, but not least, considering the companion computer (or secondary MCU) as part of the Ardupilot platform rather than an external component, so features that requires a more powerful platform than a Cortex M can also be contributed to the Ardupilot project, rather than being an individual project that only the creator (and the few people who noticed it) can benefit from.

For those who haven’t read the entire thread: I know that the current FC+CC allows you to replace any control loops by your own, but you have to replace it instead on building on the existing control loop, and the control loop you develop will be outside the Ardupilot ecosystem so the rest of the Ardupilot community won’t benefit from it (except those who stumbled upon an old post about it). By developping a distributed architecture, and as long as you segregate the fast critical loops (only on the MCU), the slow critical loops (normally on the MCU, but portable to an external platform), and the slow non-critical critical loops (indifferently on the same MCU or an external platform), you include by default features that require more powerful platform than the current MCU (once tested and approved by maintainers, of course) and you remove the hassle of combining complex features (based on OpenCV or OpenCL for example) and fast time-critical control loops on the same CPU using a non-preemptive kernel (anyone who has tried to develop control application on Linux with both time critical and complex modules will understand the pain).