Hello everyone,
While we were manually tuning the gains on the roll and pitch on a X8 copter today, we encountered two cases of « EKF primary changed » during flight.
After analyzing the corresponding logs, we found out that the first barometer glitched punctually by reporting an abnormal pressure spike forcing the change in the EKF. The way it appears in the log, as one single frame out of range, makes me believe it could be a communication problem with the barometer…
Is there a way to determine whether it is an hardware or software issue?
I do not believe it is the case since the Cube was heated, but could the low temperature outside be causing this problem?
General hardware in use :
FCU : Cube Orange
ArduPilot Version : 4.0.0 (49693540)
Carrier Board : Kore Carrier Board
GPS : 2*Here2 (One set up on I2C connected on the GPS1 serial port, the other is set up on UAVCAN)
Flight conditions:
Outside Temperature : -3,5°C (Because of the outside conditions, all of our flights were done after the FCU was heated internally, got to its targeted temperature and rebooted to avoid cold boot issues.)
Wind : Approximately 5 km/h (~10 km/h in gust)
Visibility : Clear
Below are the two corresponding logs where the problem occurs :
PS: you will see that there are a number of « EKF_YAW_RESET » reported at the end of each logs while the copter is landed. This phenomenon occurs every time we download logs from FCU using the v4.0.0. It happens for us on multiple of our copters, and occured on multiple type and versions of the Cube FCU (we do not have any experience with other types of flight controllers). I don’t know if this behavior was reported yet.
Won’t matter that your IMU is heated inside the Cube. The Here2’s each have a barometer inside, and that’s not heated. When set on I2C mode they are invisible to the FC, but in CAN mode they are, and your primary barometer is the one inside your CAN Here2.
There’s a constant, roughly 4.5m discrepancy between the two baro’s throughout the flight (first log) nothing to worry about. But when your primary baro glitches, the 2nd takes over, and that’s the EKF change.
I do understand why the EKF change took place, but what I would like to determine is the cause of the spike.
Contrary to what you said the first barometer is not inside one of the Here2. The barometer is not enabled by default in the Here2 and we did not specifically activated them. The two Barometers logged are the two onboard barometers from inside the Cube Orange. As you can see in the logs, its logged temperature (at least for the first one which glitched) follows the targeted temperature at 45°C.
The issue here is really to understand why the first barometer which inside the Cube (and heated at 45°C) did encountered a spike like that.
By the way, in the case of an external barometer, is it possible to know which barometers are connected on the FCU and get their order? The same way it is possible to obtain the compasses and the IMU IDs in the HW_ID tab in Mission Planner.
Barometer glitches are quite rare so it is slightly suspicious that it has happened on a relatively new flight controller (the CubeOrange). Still, for now I think we need to assume that it’s a one off hardware or communication error unless we see more cases of it happening. For example if we saw an increase of reports of barometers glitching across boards we would dive into the baro driver and look for issues, if we saw an increase just on the CubeOrange we would work with Hex to try and figure out the issue.
… so for now I think we should just keep an eye on it.
I’ve passed on this note to some of the people at Hex.
On the positive side, it’s good that the EKF handled the glitch well.
After posting the first time on this topic, we changed the Cube on the copter for safety reasons, assuming it was an hardware problem. There was no problem from the copter ever since, however for the record we went back to a CubeBlack (ArduCopter v4.0.0) for the time being.
I analyzed the other logs logged inside the Cube and I found out that the problem arose quite a lot before we took note of it and at a much higher rate. Contrary to what the two above logs suggested where it appeared during flight, the glitch also arose when the copter was disarmed. I found out six other logs showing the same glitch during the sensors calibrations process and when the copter was on stand-by.
Since we changed the Cube, I will keep this “faulty one” on my desk for the time being until we plan to RMA it to Hex (we are one of their resellers in France). So, if there is a need to further test this unit I can take care of it.
Thanks again for your continuous support.
Below are the six other logs showing the barometer glitches:
Here is the barometer glitching at a much higher rate before we did the compass calibration:
We’ve had another report from one of the core developers of barometer glitches also using a CubeOrange. It’s possible this is just a coincidence but we’ve reported both incidents to Hex in any case.
You posted in a Copter 3.6 thread.
You posted in a Baro Glitch thread
You posted in a EKF Primary Changed thread.
The only thread that is valid is the one you started 20 days ago where it was advised that you follow the Basic Tuning Instructions which you still haven’t done.
That’s possible that I have the same baro glitch in my Cube Black. Every time I am yawing and making circles with my copter I get EKF primary change. Please look at this thread and two logs 1and 2.
Hi everyone, I’d also like to report the same issue. I have barometer issues occurring on Cube Oranges with serial numbers CU11P21300819 and CU11P21800947. I strongly believe it’s a hardware issue and hope Cubepilot/Hex will resolve this soon.
