I have read in several posts that a 2..3 m/s reading in the airspeed while the plane is still is normal, and I have verify it myself (I don't know with which sensor, though). I use to blow the airspeed sensor to check it.
regarding the baro shift, I understand that having an airspeed sensor does not affect the baro readings, however, the baro sensor is very sensible to airflows surrounding its case (and also to ambient ligth), that is the reason that the opening of the sensor have a piece of black foam in almost all autopilots.
A few days ago, a flight of less than 5 minutes ended with a 4 meters of altitude drift, with no noticiable weather change. I think that some drift is always inevitable.
However, it could be also a flaw in the design of the autopilot. I have a "hobbyking pixhawk lite 32" that have very noisy readings of the barometer (+/- 2 meters easily with the plane on the ground). I read in some post that adding an electrolitic capacitor in the PCB could reduce the noise, and it worked for me.
Another option that I want to try is to send from mission planner periodic updates of the parameter GND_ALT_OFFSET from the reading of another autopilot sitting on the ground, to compensate for pressure drifts (it should be done automatically easily, but I cannot find any groundstation that is able to it).
Regarding the GPS-Baro difference, I am not sure, but it could be that the baro altitude is always relative to the home location when arming (zero for the home point), while the GPS could be the absolute altitude from the mean sea level (I am not sure about that, but could be the explanation). Additionally, GPS altitude have usually a large error depending on the place on earth and the number and position of the satellites.