Loiter and rtl wobbling

Attitude control and tuning is the problem.

I suspect your MOT_THST_EXPO is wrong and should be about 0.77, assuming you are not using T-Motor Flame ESC’s.

PID’s are very low, I presume from a previous Autotune or some manual tuning. Probably increase these ones for a start, only a small amount at first:
ATC_RAT_PIT_P,0.06
ATC_RAT_PIT_I,0.06
ATC_RAT_RIT_D,0.00375
ATC_RAT_RLL_P,0.06
ATC_RAT_RLL_I,0.06
ATC_RAT_RLL_D,0.00375

For safety of such a large sized quad I would set:
FENCE_ENABLE,1
FENCE_TYPE,3
You wont be able to arm and fly until there’s a valid GPS 3D fix and home can be set.

I would probably set INS_ACCEL_FILTER back to 20 and see if flight is stable enough, then set up Harmonic Notch filtering as follows:

HNOTCH phase 1
INS_LOG_BAT_MASK,1
INS_LOG_BAT_OPT,0

  • hover test in ALTHOLD for 1 minute at least, check FFT in MissionPlanner

HNOTCH phase 2
INS_HNTCH_ENABLE,1 <- set this then refresh params to see the rest
INS_HNTCH_MODE,1
INS_HNTCH_REF,{hover_thrust}
INS_HNTCH_FREQ,{peak freq from FFT}
INS_HNTCH_BW,{peak_freq / 2}
INS_HNTCH_ATT,40
INS_LOG_BAT_OPT,2

  • hover & dynamic test, check FFT results and adjust FREQ and BW

HNOTCH phase 3
INS_LOG_BAT_MASK,0
INS_LOG_BAT_OPT,0

  • no extra logging, assumes HNOTCH is working great

If you’re not sure about the Harmonic Notch setup, then post a link to a .bin file after each phase.

Once that is set up and working OK, then run a new Autotune.