Using a custom motor controller interface

Hi, I’m new to ArduPilot and the field of robotics generally. I’ve joined a project building a drone in which I am at the bottom of the learning curve. My background is in software development, with limited exposure to embedded systems.

As I understand it, a flight controller like a Pixhawk drives the drone’s motors directly via PWM signals. In my setup, we instead have a serial interface to a motor controller which allows me to set the heading, pitch, speed, etc. and it handles delivering power to the motors.

Is it feasible to use ArduPilot (or something else from the ecosystem) but replace the motor control parts with something that uses this interface? Are there examples of others who have done this? What would I need to keep in mind?

I also have some control over the interface’s protocol—right now it is custom, but if there is something a little more standardized we should be using, I could try to move over to that.

(Apologies if there is a better topic to post this in.)

ArduPilot can use a variety of protocols for actuator/motor output.
PWM is just one of them. A number of canbus and serial protocols are also supported - it sounds like you just need to add another one.
For serial, the current options are here:
I’d suggest digging through the code to see how they’re implemented