Do not increase ATC_THR_MIX_MAX to 0.9 it will make your problems even bigger…
Can you flash your ESC with newer firmware?
You are using 13.2 version(2 yrs old) and new is 14.9 i think…
here is mine settings
ARE YOU 100% YOU CALIBRATED THEM CORRECTLY?
After that my advice is to turn of #Programing by TX# in BLHeli settings…
AND CHECK AGAIN DID YOU CALIBRATE THEM PROPERLY…
Use voltage scaling for sure,but be aware when you battery is near end…
With filter,You can experiment with even lower filter settings but not higher than 10Hz…
Your copter is very overpowered…hoover at 15%-20%!!!Maybe to try add additional weight…
Also maybe your PIDs after autotune are too high,try to lower them for 15-20%(all pids equal percentage )
Also…you are using 3.4.6 instead latest beta wich is 3.5 rc7