Plane losing position and drifts away - EKF/GPS failure?

A while ago, I had a flight with my fixed wing as usual. In the beginning, everything went normal but after a while, the plane some how lost its position or the EKF spilled weird values…I am not sure why this happened. So it started drifting in a circular motion for a couple of kilometers.
Luckily I placed a second GPS and my 3g-telemetry in my plane and when it started drifting very far away, I changed the USE GPS option to use second. A few seconds later, the plane recovered position estimation and so I could save it.

I had this two times before and first I thought it was a GPS problem. So I put a new one in. But the problem appeared again.
My question is: could this be the problem of the GPS or also from the EKF2? This plane can fly for a very long time (>3 hrs, >160 km), so I was thinking if long distance missions could be the problem too?

Gear used:
Holybro Durandal with belonging (M8N) GPS
Firmware: AP4.0.7
Second GPS: ublox F9P with helical antenna


the key to this issue is the first GPS became lagged by 2.5s over reality:

this caused the EKF to stop fusing the GPS due to a bug, but it didn’t switch GPS as the GPS still was healthy. This bug has been fixed since that time, but your log is the best example I’ve seen of this issue, and I think it would be worthwhile to create an automatic test in our test suite that checks we behave correctly when this happens.

Thanks tridge,
I have never seen this issue on AP forum. I thought my setup was wrong! So this is solved in the recent Plane versions?
How do you make these great analysis and those pictures of data in your post above?

yes, I think it is solved, but prompted by your log we are writing a test for the simulator to allow us to test this case.
The log analysis is from MAVExplorer