Drone wobbles a lot in Loiter mode

Its not normal. Configure for Dshot 150, switch your motor outputs to aux 1-4 (servo 9-12). You may have to turn off Low RPM Power Protect from BLHeli suite. Try it with default setting 1st.

Iā€™m a big fan of APC props, these will fit a DJI hub

Cool, Iā€™ll give that a shot for sure! However, for the second set of props, it looks like they have a round hub which unfortunately wouldnā€™t fit my motors since the have the DJI style hub with 2 flats. Iā€™ll see if I can try and order the first set.
Thanks!

I tried that and it still seems to be more wobbly, could the bigger props help with this?

Thanks! Iā€™ll check these out! Generally, I like ordering from Amazon, but unfortunately seems like a lot of these specialty parts arenā€™t available.

Tuning will help with that.

Just the general PID tuning?

Yes. No reason to expect that the Dshot protocol would influence this.

Is there some sort of ESC calibration that I need to do with the Dshot protocol? I feel like Iā€™m missing something haha. Even with basic tuning it doesnā€™t seem to help much. I think it may just be that I need to keep working with it, but just want to make sure there is some step that Iā€™m not missing.

No calibration is required with Dshot.

Huh ok, Iā€™ll continue with the PID then, thanks!

Does it fly better in stabilize or acro mode?

I havenā€™t used acro mode. I do most of my flying in loiter mode since Iā€™m not an experienced pilot.

There is a case possibly similar to yours:

The craft oscillated in loiter and also had a low CG. PSC_VELXY_P and I values needed to be lowered to achieve good loiter performance. Maybe you could try the same (carefully). I still do wonder if itā€™s just masking the effect of an off-CG IMU, though.

It seems like it was an off CG IMU since I tried that some time ago and it didnā€™t work. But I received feedback that I should try autotune without my camera at the bottom and that should help.
Havenā€™t yet had a chance to give that a shot though. Iā€™ll try that as soon as I get a hold of my copter.

I agree that you should use 4s.