However, when I test it in stabilise it works fine for the first time, it says RTL which Is what I have set it to do.
If I test it in Loiter or RTL it says LAND then disarms after 5 seconds which would crash the quad.
I have the APM connected to the pc with a USB cable,
I am arming the APM in stabilise, going to 50% throttle then switching off the TX.
I have set it to RTL but it just goes to Land.
Please see the screenshot
Clearly says if you go into radio failsafe while within 2mtr of the home point (ie where you armed it) then it will just go into land mode. Land mode auto disarms when it detects the uav has landed so what your seeing is correct and normal.
If you want to test it properly, you will need to arm it, then move 3mtr or preferably more then switch off the radio.