RTL Altitude when Home is on the top of the high hill

Hi, I have the question about the situation I was in a week ago. My take off, Home position was on the top of the hill 200 m abowe sea level. My Waypoints wrere 100 m below my Home position and I started to loose my Data link signal. My safmode setup was to do the RTL in case that it looses that signal which started my nightmare that it Will go up for 60 m and straight to Home position which would lead to the crash into the hill 40 m below my Home position.
Is it so or there was no reason to get scared about?