Steve,
I have never used dual GPS. So can’t vouch for if that works or not.
You have a terrible oscillation in the position controller:
The highlighted values, IMO, are tuned a little “hot”. We have gone to different tuning in the rate controller, fly them mostly on feedforward, use the PID’s only for damping of the controls. Typical P values of .035 to .045 are common, D-gain values of either zero or maybe .0015 max. I gain values of 0.25 - 0.35. The system is going to like that I-gain for high-speed flight to be stable. In my experience your heli would be highly unstable in high speed flight with the pitch axis tuned the way you have it.
I would also reduce the low pass filters to 12 or 13, except on the yaw which I run up to 25 on some heli’s.
After getting the pitch axis tuned a little more stable, try it again. The position and Loiter controller in 3.6 has been problematic for helicopters compared to 3.5 and earlier, and I’ve never been 100% satisfied with it. Have gone to using Position Hold instead of Loiter for my own heli’s. I prefer the manual control in Pos Hold, then bring the heli to a stable hover manually and release the cyclic if I need GPS position holding. Even in Pos Hold if you have the heli moving, simply center the cyclic and let it brake, it will lunge back and forth like it has gone nuts. Manually bringing it to that stable hover and then gently centering the cyclic works really good and it will hold position no problem even in 20kt wind.
So I think most of this has to do with the braking action, or tuning.
So you can fiddle with the PSC velxy_p gain to see if you get better results. And fiddle with the Loiter braking and max speed settings to see if you get better results. @Leonardthall recommended to me to set the loiter speed to 3000 (~60kts) when I first started messing with the new system in 3.6 to get the best braking action. And that indeed has seemed to work the best here. I’m not 100% certain why that loiter speed setting affects the braking even at slow speed, but it seems to.
I’m working on getting this system to work with fully-articulated heads like full-size helicopters use, where the TPP of the rotor can tilt independent of the frame due to action of the coning and flapping hinges. I’m sure that will introduce some new challenges. So if the head in your Goblin has soft dampers so it acts like the teetering hinge on a Bell Jet Ranger head the system might not play well with that either. The way the system works at present is that it totally depends on sensing frame angle for an attitude input, and more advanced helicopters with larger rotors do not work this way.
If you can tie your Goblin 770 down, give it forward cyclic and get this to happen with the rotor’s TPP, the system is going to have problems flying it. The attitude controller, in its present design, only really works with very rigid dampers in the head.