Irregular behaviour when descending in RTL

I am having some trouble with my second quadcopter build. Details of the build are in my signature.

The copter has been tuned in autotune and behaves well in Stabilize.

As can bee seen in the log, it starts oscillating when descending in RTL. It also oscillates when trying to hold position in Loiter.

Are vibrations to blame (they are a bit high and can reach 4-5 sometimes), or does any PID values need tuning? I would appreciate to get any ideas about what should be my next step here.

[attachment=0]RTLtest.log.jpg[/attachment]