Loiter confused on which direction to go?

Hi

I have an odd problem. When I switch into loiter, the plane often starts going in a circle one direction and then immediately changes its mind. Often after loitering for 30 seconds or so it’ll do it again, change direction.

How do I get it to loiter in one direction only?

Best
Gareth