Need it for a stable hover.
I know about WPNAV_RADIUS - does it also control the radius above any point that sticks are released in Loiter, Land, and the final RTL stage?
Since this parameter is not working for me too well, are there other parameters I should look into?
Take a look at PID tuning, including rate, angle and position PIDs.
There is a total of 11 PID controllers, all 11 have influence on the hover accuracy in auto modes.
You are referring to PSC_POSXY_P e.t.c, correct?
What about WPNAV_RADIUS ? - Does it control the allowable XY error for hovering in Loiter when sticks are released (also in Land)?
Or is it used exclusively when flying a pre-planned mission from waypoint to waypoint?
No, as in the documentation, WPNAV_RADIUS is used to whether the WP has been hit or not.
When loitering, the vehicle tries its best to hold its position.
As @amilcarlucas said, PID and PSC-related parameters influence the loitering performance directly.
Basically, PID values are used to stabilize the vehicle on-air (achieve desired angles, accelerations etc.) while PSC parameters are used to (on top of PID) hold the vehicle in the desired position.
And then when you have it perfectly tuned the positional accuracy of GPS will be the primary factor.