Copter drifting in EKF3 mode under visual odometry and no GPS

I have been following the discussions on enabling EKF2 or EKF3 under various sensor configurations. I read that in one of the discussions EKF3 is recommended for the case of visual odometry.
I am using intel realsense T265 camera for visual odometry on my quad coipter with nvidia TX2 using the T265 pose for object detection.I am using pixhawk 2.4.8 and mission planner with copter 4.0.1
With MAVLINK, I am able to arm and takeoff with out issues. I also see that odometry is properly publishing copter POSE at 20 Hz, which is what we are aiming for. I am not using GPS.
With EKF2, It worked once nicely with the copter in autonomous flight. I notice sudden acceleration after the copter achieves hover altitude in our third trial. I am not able to achieve stable flight since.
I changed EKF2 to EKF3 by setting EKF3_ENABLE=1 AHRS_EKF_TYPE=3. In a couple of trials the sudden acceleration stopped, but the copter started drifting in hover. Are there other parameters I need to adjust? I am in the process of ETF tuning for stable flight.
Thanks so much for your help. Great discussions here.