Poshold loiter drift

Hey guys, I’m using poshold mode with optflow loiter recently, it can loiter well , but when I release the telecontroler, after braking it will drift for a short distance and then go back to braking position. Is there any method to make the drone stop at the position when I releasing the telecontroler? Any help would be much appreciated.