My hexacopter can't maintain itself when braking

ArduCopter
6

Are you flying with loiter mode?
If yes, may I know what is your LOIT_BRK_DELAY value?

If your drone is tuned, low noise and vibration, maybe this post may help you.