Position Hold not working properly

I have used Position Hold before but for whatever reason now it seems to work at higher altitude but not at lower altitude.

It seems that the Desired Altitude is stable at higher altitude in my case (35-40 meters). When I try to enable PosHold at around 15-20 meters, the Desired altitude drifts downwards even if the throttle stick is neutral. I have tried with a laser altimeter to maybe increase the altitude accuracy but with no luck. Look on the graph. The blue line shows stick position, the red one is desired altitude and the green one baro altitude.