I updated my motor mixing calculator to output the format expected by Ardupilot, to easily calculate the correct mixing for non-standard motor layouts:

https://www.iforce2d.net/mixercalc/

I actually did that a while ago but only just gave it a proper test yesterday:

“Non-standard” doesn’t mean you can put the motors anywhere you like, there are still these rules:

- equal number of CW and CCW props
- total moment arm of all motors of each rotation direction should be zero about both pitch and roll axes

… and for best results:

- place CoG at the center of lift (average position of all motors)
- keep rotational inertia in all axes as low as possible

The ‘moment arm’ thing might need a little more explanation. Add up the distances between each motor and the axis line (through CoG), using negative for one side and positive for the other. For example roll axis, if you have one motor on the left at 20cm away that would be -20, and two motors on the right each at 10cm away that would be 20, which totals to zero. This should be zero for all combinations, ie.

CW / roll

CCW / roll

CW / pitch

CCW / pitch