Drone crash in loiter mode after GSF converged

Log : https://drive.google.com/file/d/1K40L_RvrJMJfQQ54Mnv175slb2ctJwmd/view?usp=sharing

This drone was taken off in ALTHold mode and GSF was converged which made it switch to LOITER mode. It has vio (as source 2) and GSF yaw (as source 3). For some reason the drone had worse position control in Loiter mode than in Althold mode which ultimately led to a crash into a wall. Drone did not heed commands to pitch back from wall.

Please find the log attached, EKF also tried switching and failed probably because of this issue : EK3 Failure to initialize with no GPS · Issue #19072 · ArduPilot/ardupilot · GitHub

Any insights into reasons for crash would be very helpful. Thank you!