That shouldn’t matter. Mine will travel a second and then yaw facing the home point. You might need to check and see if your compass is working properly.
New result: Looks like it may be the same as you describe. My trip back to the home position is only about 50-100 feet, and it starts heading back in the same heading as it was at the time I invoked RTL. As it approached the home point, she came about. I think I understand it now.