My practice involves checking STAB, Alt Hld then Loiter for stable flight. If they aren’t right I consider it a STAB flight day and don’t fly AUTO. I’ve come back many times on another day to find great flight performance. YMMV
BTW: I’m looking forward to the forecast improvement in AC 3.3.
I just experienced the same problem with my Pixhawk after upgrading to 3.2.1. The altitude sweeps in the negative values. Unfortunately, the RTL kicked in in one case causing the copter to crash from almost 30m above. I never had this with 3.2
I tried to downgrade to 3.2 but Mission planner failed to fetch its image. Anybody knows how/where to get it?
Thanks!
Quick update to my previous posts on this topic. I have changed 2 of my Pixhawks to the Extended Kalman Filter and it has no effect on the altitude drifting issue. I just powered up one of my UAV’s to setup some waypoints and after 12 minutes, it was showing 16 feet agl. I powered down and back up and reconnected to mission planner and within 5 minutes, I was 7 feet agl so it does not look like a “warm up” period helps with the issue. I continue to land often and disarm to reset the altimeter. Really disappointed that 3d robotics will not consider this an issue worthy of investigation. Hoping someone that understands the source code will look into this and provide a solution that can be implemented in 3.3. 
Yup, plenty of alt drift on my Pixhawk hexa. APM2.5 dont. (Kalman filter turned off in pixhawk after several weird attitude mishaps and flips.)
Sent using Tapatalk
I have had similar issues on older code and copter and plane, turning down the tx power on the telemetry solved the issue! A capacitor on telemetry module may help too! Hope it helps!
The barometer is also light sensitive, in direct sunlight on the sensor (without the case) I have been able to get a 400m differance between shade and sunlight!
Might be worth checking the logs near takeoff. I had a major baro/gps alt drift building up correlated to the rising airspeed during takeoff. This drift did not go away after landing and thus was wrongly interpreted as baro drift.
The reason was large ventilation openings in the fuselage front which lead to pressure increase inside. Closing the major ones so that there are more openings in the back than in the front fixed this.