It doesn't use FBW min for either of those states. It should use "TRIM_ARSPD_CM" for both which you have set to 2500. You have THR_PASS_STAB set to zero so your transmitter inputs wouldn't cause the issue either.
Even though it was going about 26M/S when you switched from Auto to RTL, it sure looks like it thought it was going too slow. The Arspd_Err term spiked, it pitched down and applied full throttle to speed up. I don't know why though.
Which firmware version are you using? I wonder if it has a bug in this area.