Help with unstable transition from VTOL to fixed wing mode

after trying modify Q related params (up, down rate, trans ms, decel, etc…), my tilt rotor VTOL dolphin still not stable when auto RTH and landing (watch video), I think the plane can not stabilize itself after airbrake (osd show vtol overshot but I dont know what it means, even on arduplane document there is nothing about that message) (video link VTOL not stable - YouTube).

If I take control, fly fwba slowly and transition to QSTABILIZE mode or QLOITER mode the transition is super smooth (watch the end of this youtube video manual - YouTube)

Any ideas?