TLDR : Changing the value PSC_POSZ_P from 1.0 to 2.0 in Ardupilot fixed the altitude oscillations. There could be a problem with the default value.
Before the fix :
After the fix :
BEFORE doing this I followed the tuning instruction here (AltHold section) https://ardupilot.org/copter/docs/tuning-process-instructions.html and here https://ardupilot.org/copter/docs/altholdmode.html . Those two aren’t exactly the same for some reasons. And I do not have a lot thrust (hover at 68%) but the problems is not there because I fixed it.
Changing PSC_POSZ_P from 1.0 to 2.0 in Ardupilot fixed it for some reasons. NGL, this setting is wierd because it does not apear in the control loop of the documentation, Ardupilot advises to half it if there are oscilations but it was already at the minimum value (??? wrong default value I guess). Plus the oscilations seems to come from a weak P value that undercorrects, not an overshoot. So I multiplied it by 2 instead of 0.5 and it worked perfectly.
It realy seems like this value is wrong by default because the doc (almost) does not talk about it (only in one of two links I sent), from the doc it seems like it’s not influenced by the thrust or weight and the doc only advises to reduce it but it’s already at the minimum.