RTL mode on channel 5 does not work

My copter crashed after I switched to RTL mode using channel 5 switch.
I had AltHold, RTL, and Stablize modes assigned to that switch. In MP they all showed that they work fine.
When in flight I flipped the switch to RTL it did not respond. In fact, instead of lowering the altitude to RTL_ALT it increased it and would not come back to the launch position. I waited for a few seconds and flipped channel 7 to Land mode. Did not do it either.
After a few more seconds it went ballistic and hit the ground.
Log files show that the switch to RTL and then to Land were recognized. So why the RTL and Land modes were not executed?

Post a link to the .bin log file.

Link to log file:

https://drive.google.com/file/d/1SjBPXhzPiBl26OSN3p6MFya_Bff2gUyL/view?usp=sharing

What type of craft is this?

Single copter. Frame_Class 8.
RTL and Land should be executed regardless of type of copter.
In the past RTL worked fine in this copter but it was always triggered by battery low voltage. I never used switches for that. This was the first time and the first failure.

Just having a look through your log file. Here are some of the things I found. Just throwing out a few ideas as to what it might be caused by.

It looks as though the GPS had a glitch before you went into RTL. That could have caused the odd behaviour you seen in RTL (increase in alt and not returning home). Not sure it would have caused the instability seen in your attitude though.

It also appears as though you have had a battery failsafe early on. Was your battery monitor setup correctly? Is the low battery warning valid? If the battery ran very low this could resulted in aircraft struggling to maintain attitude.

I have never set up a single copter before so perhaps I am missing something. I noticed from your params that the servo function you have set for the RC7 is 39 which correlates to motor 7. Looking at the wiki it should be set to 37 for motor 5. Am incorrect here? Not sure what effect motor 7 output would have had. It looks like you are using quite an old version of copter so perhaps I am misinformed as the wiki may no longer be valid for that version of the code.

Looking at the board voltage at the time of the event it looks as though the average voltage increased. Could indicate a power issue of some kind? Not sure, just throwing out ideas.