MANUAL_CONTROL in SITL - vehicle falls once I stop sending messages

Hi, I am working on a custom ground-station SW in python using pymavlink.

I want to implement manual flying over the keyboard, similar to the joystick function in MissoinPlanner.

I was trying to use the SET_ATTITUDE_TARGET, but the vehicle drifts as it does not have a GPS lock (e.g. when only yawing).

Now I am trying to use MANUAL_CONTROL, but I discovered I have to switch to LOITER mode in order to enable horizontal movements.

But once I switch to LOITER, the vehicle starts descending/falling quickly, so now I am using throttle midpoint 500 in the message (as shown here) and sending the message at ~10Hz.

But then I have to keep sending the messages with throttle~=500. But this does not seem to be safe at all, because once I kill the ground-station in LOITER mode, the vehicle stays in LOITER and falls.

So, is this the correct approach or am I missing something?
Thank you,
Jarda