Tri-Copter VTOL Unstable transition back to Quad

Hello,
I have a Tri-Copter VTOL, of around 9 kg. The transitions from QLOITER to FBWB are smooth and I faced no issue. I then switched the plane to LOITER, and after a few a circuits when I transitioned back to QRTL, the transition was very unstable. The drone pitched and rolled before stabilizing at the Q_RTL_ALT. Could someone take a look at the log file and see if there is anyway to make the transition back to quad smoother? I have seen on the forum that manual transitions seem to be smoother but I don’t have that option. The plane flies well in LOITER and is also stable in QLOITER, with some yaw tuning desired.
My initial thoughts were to reduce Q_TRANS_DECEL and the Q_TILT_RATE_UP and Q_TILT_RATE_DN, and also increase the RTL_RADIUS (100 m as of now).

Log:

tldr: I need smoother transition back to quad mode from fixed wing.