If it is gyro drift, it is often related to the temperature of the IMU package. Some IMU’s are more sensitive than others to temperature variation. Can you preform a “PREFLIGHT_CALIBRATION” using the Mission Planner actions menu the next time the error occurs? This will re-calibrate the gyros to zero, just as happens during the boot process.
If the IMUs are unusually temperature sensitive there are a few options. One would be to replace the flight controller for another one that is not as sensitive. Another option, if the flight controller has an IMU heater like a Cube flight controller, you can preheat the IMU package by powering on a few minutes before flight. Then preform the preflight_calibration action or a reboot once it has reached operating temperature. If it does not have a heater you could try adding some amount of insulation to the flight controller compartment to reduce the temperature changes during flight caused by the airflow.
If you can provide a dataflash log of the event with log_disarmed enabled that could help identify the problem.
I too though it may have to do with Gyro temp so I tried to do some tests insulating the flight controller (and then removing the insulation). There seems to be no correlation between IMU temp consistency and to the issue unfortunately.
It seems that longer and faster I fly, the greater the “-” variance value is about 60 sec after I land.
The graph below depicts the variance issue. What does NKF8 IYAW actually mean though? Does this have to do with different compass variations? Does it have to do with gyro variations?
The warning is given when the variance the value seems to go over -/+20deg.
Here is the log: https://www.dropbox.com/s/069qx245i3y4qbz/log_15_2020-5-9-18-27-18.bin?dl=1
Looking at the log those gyros look healthy and temperature does not appear to be a factor. However the NKF8.GSZ (z-axis gyro scale factor of the second EKF2 instance) does have some rather large values. The second compass has some unusual drift in readings of the Z axis compared to the others as well.
NKF8.IYAW is vehicle yaw innovations in the second EKF2 instance. My understanding of the EKF is pretty surface level but I believe these values are essentially the size of the error in the EKF prediction that is fed back in as a correction. The ArduPlane wiki has a good list of dataflash message meanings. Many apply to copter like the EKF information. https://ardupilot.org/plane/docs/logmessages.html For messages about the EKF2 instances NKF1-4 is for the first instance, NKF6-9 are for the second instance.
We had the problem too, as mentioned before, after switching the cube on two drones, the problem did not occur anymore after many testflights.
After spending way to much time on this I finally figured it out. CM, like you point out, the issue in my case is Gyro drift on the Z-axis on IMU1. It’s a subtle drift at about 0.5/sec after I land but this is enough to create an EKF2 warning. Over 60 sec of time this cause 20+ deg of drift deviation. Anyways a work around for me to set EK2_IMU_MASK so that I use IMU 0 and IMU 2 (instead of IMU 0 and 1) and all my warnings are gone. I tried flying on IMU1 alone and it still works (although it’s noisy), but it just drifts to much to use in EK2.
Thanks for all the help!
Hi there, sorry to be absent for so long on this issue.
We’re preparing to release Copter-4.0.4-rc1 for beta testing this week and I’d like to ensure we get the right fix (if necessary) included.
The basic check is that the two EKF cores and DCM are all reporting the same attitude. I think we have a couple of different failures reported above:
- “EKF2 Yaw inconsistent by XX deg” means that the two EKF cores are reporting different headings which is quite bad and we should try and resolve this. It seems like from the analysis above that some IMUs are suffering from high z-axis gyro drift. I would have though that performing a gyro calibration would resolve this.
- “DCM Yaw inconsistent by XX deg” means the DCM doesn’t agree with the EKF. This is probably a false-positive and I’m planning to disable the check of DCM because we don’t actually use it in copter. It runs in the background but the flight code never uses it for attitude control or anything except reporting.
So if you’re seeing these types of errors:
- if it starts with “DCM Yaw” please post a log and I’ll have a look to confirm that the EKF is OK and we can disable the DCM check in 4.0.4
- if it starts with “EKF Yaw” then please try triggering a gyro recalibration and see if that fixes it. If not then please post a log and I’ll have a peek.
- I don’t think it’s a good idea to switch to the EKF3 to try and get around this problem. The EKF3 seems safe but we know there are some small issues. By the way, we plan to make EKF3 the default for Copter-4.1.0.
@Bobmarr kindly provided logs above. I’ve had a look and in the 11.BIN log it looks like DCM is off, in the 12.BIN log it appears the 2nd EKF lane is off. I’ll discuss this with Tridge and others.
Thanks very much for the log and sorry for the delay on this!
If it helps, here is a logfile from a flight today, after calibration, with the same problem:
@rmackay9 I have this Yaw Inconsistent as well.
I have this problem ek2 yaw inconsistent by (x) degres (always different degrees), but in my case it is in Arduplane, with PH4 and PH4 Mini, I am using the latest version of firware, but loading previous versions the problem persists, problems that for true before they did not exist.
I’ve been reading this for days and trying all the multirotor solutions on my plane and nothing works. At the beginning of the year I had this problem in multirotors and I wrote to Holybro to find a solution, they told me to load the firmware manually, but it was not solved, suddenly one day the problem ceased to exist and I have never had it in multirotor again. . But this problem continues in Plane now. Someone can think of why this happens, in my case before in multi and just not now but in plane?
In PX4 firmware doesn’t have this problem.
Kind regards to all
Did you re-calibrate the magnetometer? Outside in an open field away from metal structures and with a GPS Fix?
no, but reading all about this, it looks its an IMU problem not the compass, Am i right or not?
Hello, I I just discovered something with this problem of ek2 yaw inconsistent, if the controller points north the problem does not appear,and I can arm, and the degrees of inconsistency are the degrees expressed in the compass, that’s why I always see a different number.
I do not know if this has been discussed in this post. So if I arm the plane whit 0 degres top the north there is not problem, looks like its going to be ok, but if disarm and try to arm in an other way whit no 0 degrees the mesage appears .
so is the solution arm pionting to the north? or going to have problems flying?
I think I’m a little late in this post, but I’m experiencing pre-arm errors “EKF2 Yaw inconsistent” identical to those described in this post. I’m running AC 4.0.3 in a CubeBlack, I attach logs and graphs of the errors, if still useful.
Binary log files
Could someone tell me if there is already a solution to this problem or if this is already fixed in AC 4.0.4-rc1??
Yesterday I installed version 4.0.4-rc1 on this drone and I made some bench tests on it. This problem regarding the “EKF2 Yaw inconsistent” persists. I attach logs and graphs of the new errors in 4.0.4-rc1.
@rmackay9 Have there been any fixes to this problem in 4.0.4-rc1 as you said earlier in this post?
On the other hand, I’ve been looking into the logs a little more thoroughly to see if I can get anything more out of them. The first thing I noticed is that the reading of IMU.GyrZ from the IMU number 2 of the CubeBlack begin to drift slightly, reaching a maximum difference of 0.1 with respect to the IMU 1 and 2 in a period of about 9 minutes. Then I tried disabling IMU number 2, recalibrating the gyros and doing more tests. In these new tests with only the IMU number 1 and 3, the problem is completely solved and the output of the EKF no longer differs.
Later I realized that this deviation from the IMU2.GyrZ reading correlates perfectly with the change in temperature of the IMU while it is heating up. Furthermore, in some cases the temperature does not remain constant when it reaches the target temperature of 45ºC and continues to increase slightly over time.
Any idea how we can solve this problem?
I leave here a link to all the LOGs and captures I have used in these tests: https://drive.google.com/drive/folders/19fSzhGjTsZ9sElJWpx0gpcZLffO5pWCq?usp=sharing
Maybe we need IMU temperature compensation. Someone posted about it in a similar IMU drifting thread yesterday.
This would be interesting if it was added to the AP code. We have environmental chambers we use for temp comp of load sensors I could possible get some time on. Huge backlog now though as we just opened that facility back up. And a Kakute F7 and Omni Nano F4 on the bench doing nothing for test.
I have a freezer and heater and plastic bag w/ silica gel :). That should still get the task completed.
I don’t think we could share calibrations. If temperature is causing the issues I’m seeing, each IMU has its own range of being affected by temperature. Because for my leaning issue, basically, some omnibus nanos are much worse than others. Some of my copters get this Yaw Inconsistent message every time I power up, while others are rare, on basically identical vehicles.
Yes, my experience also. I thought I damaged my Omni Nano F4 from a crash and bought another one only to find the same drift problem. And it was worse.
Yes, a feature like that will be very useful in AP to deal with this type of situations. I know that this type of beta-flight boards (Kakute, Omnibus, etc) didn’t perform well in terms of sensor stability, but for the price they do their job very well in most cases.
But now talking about a controller like a CubeBlack. You think this problem is purely hardware related? Is there anything that can be done right now to compensate or calibrate this bias?
I’ve used a multitude of CubeBlack flight controllers and I’ve never come across a case with so many gyro inconsistent errors……
Other than adjusting bias learning, EK2(3)_A(G)BIAS_P_NSE, I don’t think we have other parameters to adjust. And I’m not sure those really help. It does change the rate the bias grows in the logs, but the symptoms I was having didn’t improve.
One thing I’ve always wondered is if the more expensive FC manufacturers test each IMU and throw out (or resell) the ones that are out of spec. If so, and yours is not performing well, maybe they will RMA it. It’s probably worth asking, if you know this one is definitely performing worse than other CubeBlack’s.
From my experience with the cheap boards, it seems to be hardware. They all function at the basic level of reporting angles and velocities. But some constantly build the bias quickly while others don’t. Some lean badly and others don’t. This is testing the same brand FC on the same vehicle, so not many other variables.