Do you know the OBAL system? It implements quite some of the functionality you want already (except additional UARTs), and proove already that it is actually working on a pi zero hard- and sofwarewise: