Well first things first, compass calibration needs to be done outdoors, not inside a building. It needs to be done away from structures, vehicles, reinforced concrete slabs, steel fencing, and other sources of magnetic interference. And you need to not have sources of magnetic interference on your body, like magnetic jewelry and accessories.
Compass calibrations does not need to redone at every new flying site. It only needs to be redone if you've reconfigured the copter, added/removed hardware, done a parameter reset, or it otherwise asks for it. Or if you didn't do it right to begin with . You should definitely NOT be redoing it every time you go somewhere else to fly.
Also make sure you did the accel calibration correctly. It needs to be perfectly still in each of the prompted orientations when you press the key. So it needs to be done on a solid and stable surface like a floor or table. You can't just hold it up in the air on it's side.
Beyond those things, vibration will ruin your loiter. I can't open the logs at work. But that's the next thing to look at in the logs after you have done your calibrations correctly!