RTL does not RTL but does larger circles at RTL altitude

I was trying to fly a mission where a DJI F550 frame with a pixhawk would takeoff and fly sideways using the do condition yaw function. It worked once and I landed and adjusted the speed on the mission re-uploaded it to the F550 and then did an auto mission again. It flew to the far waypoint and then started making small circles trying to stay on course. I tried to take control back with the remote and hit RTL. Then it started doing much larger circles at the same altitude. Since I was beside it and it was not responding I grabbed the landing gear and pulled the battery.

Any help would

Could you also attach the .bin or .log dataflash file. There is more information given there than the TLOG file.

The first thing I would do is recalibrate the compass and the IMU. Is your GPS and Pixhawk pointing in the same direction and pointing forward? Which version of Arducopter are you using? I just upgraded my Hexacopter to 3.2 and had to redo the calibrations.

When connected either by 3DR telemetry radios or by usb to the pixhawk I clicked on dataflash logs -> Download via mavlink -> and get an error cannot get log list.

I am running Arducopter3.2 (c830f3e1) on a hexa x frame.

This is pulled from the cf card in the pixhawk. I caught the Hexacopter as it passed me and pulled the battery which is why it ends in the air.