ouch! I'm very glad it survived.
Doing your first transition from QLOITER to LOITER is certainly "courageous". I should explain what it tried to do.
As soon as it went into LOITER it wanted to change from a hover to flying in circles as a fixed wing. That means the following things happened:
* the nav controller told the fixed wing controller to bank hard left (it demanded a 25 degree left bank)
* the multicopter controller saw the demanded left bank, and tried to help the left bank with its attitude controller
* the navigation controller told the multicopter code to yaw hard as it tried to line up to start circling
* the tilt servo started tilting forward
It should have been possible, but I don't believe anyone has tried this before on an ArduPilot tiltrotor. For first transition we recommend FBWA mode as it is the simplest mode, and you want things as simple as possible so you can separate out any issues. Flying straight ahead in FBWA mode is the simplest possible thing, then once you have good airspeed and have finished your transition do your first turn. Be ready to go to QHOVER if something goes wrong.
You certainly did the right thing by climbing to 90m for your first transition (you said 100ft, but it looks like you did 90m, which is good!). You lost about 50m rapidly when things went wrong. Also well done for flicking back to QLOITER quickly. I would suggest that QHOVER is a bit better "recovery mode", as it is simpler, but QLOITER is ok if you don't have QHOVER handy, and it clearly worked.
The problem now is working out exactly what went wrong. I have plenty of data in the log, but the data is all mixed together as its trying to do everything at once. Every controller is working hard, and clearly overshooting.
The most interesting part of your log is actually the QRTL part as that is showing attitude control issues in pitch, and is much simpler as it isn't transitioning.
Let's look at that QRTL. You were in QLOITER and changed to QRTL at about 38m. The log shows the plane wanted to turn right, put the nose down and descend. This is what happened to attitude:
(sorry for the "UNKNOWN" that is QLOITER, its because I trimmed the log for rapid analysis)
You can see that the right roll succeeded as Roll and DesRoll track each other OK. The pitch is much less happy. It does initially put the nose down a bit, then the nose comes back up while its trying to put the nose even further down.
Let's see what happens with the motor outputs (its a multicopter at this point, so the only control it has is with the motors)
You can see that motor 1 and 4 saturate at max power. They are the two rear motors. So it was really trying as hard as it could to stick the nose down, yet the nose came up. Let's look at the pitch PID controller:
see the PIQP.I ? that means that when it was in QLOITER it was having to apply more power on the rear motors. It is either tail heavy as a multicopter, or the rear motors are not generating their full power level. You can see that in the motor graphs above too.
I'd ask you to do a few things. First off, go over those rear motors and props carefully. Check you didn't do what I did when I first flew my FF6 which was to have one of the props upside down! (writing should point up on all props). Does everything look ok?
Next, use the motor test to run each of the motors one at a time at 70% throttle. Do they all sound the same? Do either of the rear motors sound any different?
Next, how far off the recommended CoG are you? Can you put a bit more weight in the front?
If all the hardware seems good and motor test is good, then next step is to go back to QHOVER. Flying in QHOVER mode at a reasonable height, can you demand both right roll and down-pitch at the same time and it responds correctly? Does everything sound OK when you do it?