EKF variance error

We are working on GPS denied Navigation and the odometry and path looks consistent (no drift) as visualized in rviz using rosbags recorded during actual flight. But when we send the vision position estimate and vision velocity estimate to FC, it sometimes shows error variance in position and / or velocity.
In our vision position estimate and vision velocity estimate, we tried estimating the covariance of delta position and velocity as a sliding window of N odometry messages (tried multiple values of N from 3 to 30) and variance mostly stayed below 0.2 rarely spiking upto 0.6. But regardless of what pose covariance and velocity covariance we send, FC’s EKF shows variance error multiple times.
We are using Jetson nano with Realsense D455 for VIO.