Stabilise is definitely a handful on some multirotors, but it is the go-to mode if something goes wrong - you have control.
AltHold plays much nicer with the throttle being a “fly higher / fly lower” action rather than bare-bones throttle.
Loiter is the sanitised GPS assisted mode that you would normally spend most of you time with. You can let go of the sticks and hold position against wind, or move precisely.
On your transmitter set a 3 position switch for Stabalise, AltHold, Loiter
I find it handy to set a channel and switch for killing the motors → RC13_OPTION,31 (channel 13 in my case)
For the next flights use AltHold
Try to have the hex hover for a while and just do some gentle pitch/roll/yaw movements.
Set these mostly to gather data, but it should help to smooth things out a bit too
INS_ACCEL_FILTER,10 (this should be the default value here, it will help)
INS_LOG_BAT_MASK,1 (gather data for the Harmonic Notch Filter)
INS_HNTCH_ENABLE,1 ← set this then refresh params to see the rest
INS_HNTCH_REF,0.15 (a guess)
INS_HNTCH_FREQ,60 (a guess)
There’s some vibrations which will affect flight and tuning. It’s not to critical levels, but not the best either.
First check nothing is touching the flight controller or rubbing against it, wires are not too loose or pulled too tight. Secure everything to the frame.
I guess you have the Pixhawk mounted on the 4 foam blocks it came with? Maybe try to add some weight to it, I stick a flattened piece of lead to the underside. Otherwise you have to find softer mounting foam (or gel pads) or change the mount entirely to something like this:
Vibration Damping — Copter documentation
You can see in the graph that vibrations go up quite a bit at some stages. High vibrations, particularly Z axis, will cause control problems and potentially a fly-away.
If you can tame the vibrations a bit and put in those other settings, let’s see the new log file and you can progress hopefully to Autotune.
During those test flights, are you meaning to operate a channel for a gripper? That’s OK if it’s intended, just curious