Try to change according to this expert suggested values and see if it improve. Particularly the LOIT_BRK_DELAY.
Also your post title says yaw, then your content said loiter speed forward…
Could be your drone having high noise or high vibration without looking at your log file.