Bad reaction to controller in Loiter mode, other modes fine

A few days ago, I did a successful test flight with a new build in loiter mode (Pixhawk 1, Arducopter 4.2, Holybro X500 V2 frame). However, I observed some shaking during flight, so I reduced some parameters by 50% according to this. I also performed a manual ESC calibration since the throttle went up a little late.
From this point on, the motors reacted very slowly and non-linearly in loiter mode. In stabilize, the behaviour seems to be fine. I completely reset the FC, but that didn’t fix it either.
Since I’m a really bad pilot and loiter mode is the only mode I can fly without crashing, I pretty much depend on it…

The following videos illustrate the behaviours:

Loiter:

Note that this happens inside on my desk with questionable GPS accuracy, but the behavior is the same as outside.

Stabilize:

The log of my failed loiter attempt is here.

If you are basing this on the bench test in those videos this is expected behavior. Inside/outside won’t matter. Use Mission Planners Motor Test to bench test motors.

It’s not based on the bench test. I attempted a flight in loiter mode outside with good GPS reception and immediately experienced this behaviour. The bench test was done later to illustrate the problem and it behaves just like outside.

The motor test under Setup->Optional Hardware->Motor Test works without problems. Should I look out for anything there?

Post a link to that .bin flight log.

It’s the log I posted at the end of my question. Because the motors where so uncontrollable, I never lifted off. That’s why I call it a loiter attempt.

Yea, that log is of no real value other than to extract parameters from. I think most get those off the ground with the S500 parameters posted and tune from there.

Why is the log of no value? What else can I provide or do to help this discussion.
By S500 parameters posted, you mean going to “Config->Full Parameter List”, and then choose Holybro-S500.param from the dropdown on the right? I did that, excluding the BATT_ values since they’re assume a 3s battery instead of 4s. However, this did not change my original problem of sluggish loiter mode behaviour.
With all due respect, I appreciate your help, but you’re being a little cryptic in your answers.

It doesn’t leave the ground. It’s not clear why motors 1&3 (front motors) are being commanded higher. Post a log of it flying in Stabilize mode.

I’ve recently set up an X500V2. I didn’t use the default parameters in the configuration drop down list. I’ve had mixed luck with those. If they include PID values for a different battery or prop/motor then what you’re using it will only cause you trouble.

Rather, I used the initial parameter setup, and then followed the normal process of filter tuning and auto-tune. Two batteries later and I was done.

Okay, one stabilize mode log:
https://www.dropbox.com/s/u954boc6vm6gxil/2022-09-09%2008-40-18.bin?dl=0

And yes, it consists of a series of short flights that lead to crashes within a few seconds. I’m afraid, that’s ultimatively the best I can do. As soon as I lift of, It begins drifting in random directions and I simply don’t have the ability to hold It in place.

Btw.: I tried AltHold mode once and observed the same behaviour as in Loiter mode.

Back to my original problem: I replaced my Pixhawk with a spare one and noticed the same behaviour in Loiter mode. I’d most problably pin it on the ESCs, since the problem seemed to show up after manual ESC calibration.
Holybro doesn’t give much info about the ESCs in this kit, but they seem to use BLHeli S firmware.

Use the Semi-Automatic ESC calibration method
https://ardupilot.org/copter/docs/esc-calibration.html#semi-automatic-esc-by-esc-calibration

Set this
INS_LOG_BAT_MASK,7
and you should be able to use these:
ATC_ACCEL_R_MAX,100000
ATC_ACCEL_P_MAX,100000
ATC_ACCEL_Y_MAX,20000
ATC_ANG_RLL_P,8
ATC_ANG_PIT_P,8
ATC_ANG_YAW_P,5.5

Attempt a take off and send that log