Oops! I have figured out the cause of the issue - rookie mistake! ...which as I am not a rookie should have never happened! Turns out that my radio was commanding left aileron, hence the drift left, and loiter...well its possible there is a bigger deadband in loiter so it wasn't effecting a drift due to that. I replaced my radio recently and even installed M9 gimbals in it, and even though I did a radio calibration on the Taranis, I forgot to check the radio calibration section in Mission planner - of course we need to ensure that center stick reads 1500 for each model... and mine didn't! So I tweaked the sub-trims in my model outputs screen to get 1500 on A,E,R centre sticks in MP and all appears good now thankfully.
To be honest, even though I don't have the commanded drifting issue now, the hexacopter in particular feels very different on 3.5. Its far more responsive to stick inputs. I reduced Stab P values down from 8 to 6 and also reduced RC-FEEL to 0.5 from 0.6, but still super sensitive. Think I need to run an autotune if we get a low wind day.