In most time, RTL and Land modes always work fine for me. But today, when I switched mode from Loiter to RTL (throttle was middle up), the drone flied back, but kept ascending. I tried to lower it by lowered the throttle, switched mode to Land, it not worked…
Finally, I succeeded switching to Loiter mode, lowered the drone, and finally, Land mode worked and it landed.
I checked your log, and found your viberation is too high, very possibly this is the cause of the ascending.
you can reduce the the viberation according to arducopter wiki.