Drone Loiter mode drifting

Hi, I’m relatively new with autonomous drones and I hit a dead end with a bigger drone.
I was testing it, approximately 20kg weight UAV. I use the PixHawk Orange Cube+ as my FC and HereLink as my RC link. I also have a Here3 as my external GPS and compass combo. The drone has 4 hobbywing 100kV motor powered by a 12S 16 AH battery. The altitude holding is done by a downward facing Lidar.
My problem: I take off in altitude Hold mode using the herelink RC with QGC on it. After, I switch to Loiter mode, the drone becomes instable, and I must switch back to avoid bigger oscillations and sudden horizontal movements.
I did 2 tests with ardupilot version 4.3.2 and 4.3.6, both the same result. The parameters list was given with the drone and is said to be able to use Loiter flight mode normally. (Stable horizontal position holding, no over compensated breaking and no oscillating angle changes.) I tried calibrating the compass and the accelerometer, but no change.
So far, I also looked at the logs from the flights, and checked the desired Roll, Pitch against the measured Roll and Pitch, and it was fairly in sync. Is it possible that with larger crafts the small Pitch-desiredPitch errors have larger impact on the flight characteristics?

I plan to test 3 things ASAP:

  1. FFT resonance test with Mission Planner. Can the frame make such oscillations, that the current filter setup is not able to handle it?

  2. Calibrate the compass with Mission Planner to counter the ESC and motor interference.

  3. Use MP autotune for the drone. I’m not sure the issue is the PID values, because the follow error is small.
    Did I miss some obvious thing in setup or the parameter list? Or is it simply shaking itself apart midair?
    Thanks for the help!

Param and logs:

Before you advance any further you need to enable EK3, disable EK2 and start from there.
Set these and them make another hover flight in AltHold as you need to configure the Notch filter before running Auto Tune again:
INS_LOG_BAT_MASK,1
INS_LOG_BAT_OPT,4
PSC_ACCZ_I,0.40
PSC_ACCZ_P,0.20