Servo starts jittering, when gps fix is bad or there is no gps fix

Hello, when there is bad gps fix I have a very strange behaviour, the plane starts to have massive oscillations on pitch axis. This is not pids as I see it, because when the gps has enough sattelites the wing flies perfectly well. It looks like gps has influence on attitude estimation. If I set AHRS_GPS_USE to 0 could it solve the issue? Sorry, but there is no log, I can record it next time.

@Yarillo
Maybe a good idea is to tell more about your setup and mainly in which mode you are flying?

As you can see on the osd it was FBWA, however such thing also occured in STAB, RTL, cruise.
It is matek f405wing v2 on custom ardupilot firmware for ek2 use on f405 mcu. The plane is a 3d printed wing.