CubeOrange main thread block 500ms

Hello! I’m currently running c75848435eeac4ee75089a488e46a73ef8c4a89e off master Dec 21,2020. On Cube Orange Hardware. During a flight the main loop seems to have frozen up for 500ms, and immediately afterwards the EKF variance blows up. Is this something that is known and fixed in upstream? If not what next steps to debug this would you take?!


ekf|690x351

dataflash log and tlog would be great if you can get them.

Of particular interest in the dataflash log is the MON message.

Hi Peter I might be able to share the log offline via discord. Sadly the MON message does not exist on this version of 4.1.dev.

I’m going to take a random stab in the dark and say that it could be logging related. It’s outside my area of expertise (PeterB is the expert) but there have been some rare cases of the main loop stalling for a moment because of logging but I think these are all fixed in master though.

Did you see the ekf graph? It blew up like crazy right after the log cuts out which made me think it wasn’t purely a logging artifact

It caused bad variance failsafe to trigger.

If the main loops stalls though then the EKF becomes unhealthy because it ends up missing a bunch of sensor data and the sudden jump in new sensor data doesn’t match it’s prediction based on the old data.

1 Like

Awesome! I will check the upstream for anything where the main loop got stuck from logging. If you know any fixes off the top of your head let me know. But I think you’re right that this is probably the cause

1 Like

@stevieVan1,

This PR from Tridge is the most recent that I’m aware of but I think there were others done by PeterB.

1 Like