Hi, I have an Protos 500 and I still have simmilar problems as you do.
at first I would recommend to get rid of using the EKF stuff....
Bulit many drones and now 2 helis ....cannot see the benefit of these ekf values.
Okay your gps is mounted on the tail ? (mine is)
I must say that we cannot adjust the compass MOT setup fot traditional helis ...cause it is dangerous to spool up the heli while connected to your laptop .
Do you run govenor mode ??
I programmed 3 different RPM levels and I observe that the faster the rotation, the more stable it is ....only in Poshold mode I observe the swining in pitch and roll.
I am not sure to increase or decrease the levels .
In Stabilize mode it glies very good...what effect is your heli showing ?