Please, help me to find the reason of my crash. Hexacopter DJI S900. Pixhack V3 with AC 3.6.5
It was second fly. First time it was ok, but Gyros inconsistent message appeared few times.
I swiched IMU Pre-arm check off and made another flight in several days.
Second fly was ok at first, but than I turned copter around and tried to go Loiter mode. After it has made hard movement in roll axis and after second made a flip and turning around pitch axis went to the ground.
You disabled many important arming checks, INS,GPS , took off with an mediocre GNSS fix, ignored Bad variance warnings , GPS Glitch, EKF variance.
Was this outcome really unexpected ?
then your motor5 failed to produce thrust, and you had not enough thrust to deal with that.
A GPS flight mode was not even being used. The aircraft took off in Stabilize and switched to Alt Hold. The GPS Status was 3 thruout the whole affair. The real problem is the EKF. It decided there’s a GPS Glitch (which can be caused by a number of reasons) and it switched processes. The two EKF processes do not agree on attitude at all.
The obvious outcome is a crash when it switches the EKF processes because it causes a radical upset in the aircraft’s pitch and roll attitudes.
Apparently, you do need to read up on how EKF works, and what it does.
It does not magically stop to gather and correlate data just because you are not using an “gps mode”
in any case, the fault was not directly caused by EKF trouble.
The roll attitude had diverged 25 degrees and pitch by 23 degrees between the two EKF processes before it logged the EKF Variance and subsequent GPS Glitch. The DCM had also diverged.
An attempt was made to switch to Loiter but was refused due to the EKF. The EKF (wrongly) switched processes and crashed the aircraft. The aircraft was already airborne before the first error was logged.
So I’d be interested to hear the solution. As I’ve spent great deal of time arriving at a fix for helicopters so the EKF doesn’t do this.
I’d like to add that the divergence in the roll attitude alone went to 100+ degrees when it switched EKF processes. There is no way to stabilize the aircraft when it calls for a sudden difference in roll of 100+ degrees.
Andre, if you look closely, the EKF switched to a bad attitude solution. It was using the second EKF at the time of the flip. It THINKS the aircraft had gone into a 121 deg right roll at 30 deg nose down. But this was not the actual attitude of the aircraft. Note what the other EKF says.
The DCM and mag y output agrees with the first EKF. It had mistakenly throttled motor 5 to full throttle to counter what it THOUGHT was a bad right nose down and right roll. But the airframe was actually now flipping to the left.
Unfortunately the two EKF’s had diverged, it picked the wrong one.
Starting out from approx 25-30 deg of roll the second EKF said it rolled right, the first one said it rolled left. The first one was the correct one. The DCM agrees about the direction of roll, but it had also diverged.
Even after the horrendous wreck (it looks like it landed right side up), the second EKF thought it was at 71 deg left roll. Then a few seconds later it switched EKF’s again and now everything is great.
The attitude control is not going to be correct when the EKF is going the wrong way.
Why the gyro’s did that, I do not know. And why the EKF picked the bad one I do not know. The gyro outputs look fine from arming right up to that point.
Two of the gyros agree with each other. The second one that the second EKF process was using does not agree.
The V3 uses an Invensense MPU6000 in the #2 gyro/accel slot and these are very good low-noise gyros with built-in aliasing filtering. I’ve found them to be much lower noise than the MPU9250’s, although the MPU9250 is used in slot 3 in the V3 controller.
I think I would look at the calibration for the accel’s. If you have been getting error messages previously about inconsistent gyros it could be one is either bad, or has bad calibration. For it to go inverted that like I would suspect the gyro has failed.
Thanks very much to all for the detailed investigation above.
It looks like maybe the the orientation of the 2nd IMU’s gyro is incorrect. It seems to be rotated 90 degrees. This can be seen by graphing IMU.GyrX vs IMU2.GyrY.
@KbGo, could you tell us exactly which Pixhack board you’re using?
As a side note, the compass is suffering from quite a lot of interference from the motors. In the graph below we can see it’s length grows from about 300 to over 1000 which means the interference is far stronger than the Earth’s magnetic field. I don’t know all the details of how the active EKF is selected but the huge difference in yaw of the two EKFs (perhaps caused by the bad compass data) may have contributed to the mistake.
Too little too late perhaps but we have a new pre-arm check in Copter-3.7 that would have caught the attitude inconsistency between the EKFs before takeoff (they were off by >100 degrees in yaw).
EDIT: the arming check’s for the IMU were turned off (as mentioned above) but if left on they would have (and probably did before being turned off) cause an “inconsistent gyro” message to appear. This was a real problem and so I guess one lesson here is those arming checks actually work!
That log does indeed show the 2nd gyro has the wrong orientation. I have a Pixhackv3 here as well, so I loaded the same copter version (3.6.5) and loaded the parameters from the log, and mine doesn’t show the issue.
Can you try power-cycling a few times and see if you always get the issue?
Otherwise I think you have a bad board, though in a very strange way
Thanks a lot for your investigation. Your proofs seem to mee very close to the real things…
@rmackay9, I use Pixhack V3, not the last version (X from september 2018), but first version.
I have 4 hexacopters like this and 5 Pixhack V3 boards. One ofe them has Bad Gyro health message always and was not used at the very beginning. Also this board we are talking about has Gyros Inconsistent message. But I had a risk to use it on the aircraft.
Now I removed it from aircraft and recalibrated several times on the table. Nothing changes.
I’m affraid for other aircrafts. They doesnt have Gyros Inconsitent message, but one of them denied to calibrate external compass and only internal one is used there.
Although this is undoubtably a hardware issue I have to say I have never had good results from EKF switching. Running two or more EKFs is supposed to help when sensors/estimates get bad, but what tends to happen is that one bad set of estimates is exchanged for a different bad set but with the added surprise factor for the pilot (which can easily result in a crash - exactly the opposite of what was intended). On all 3 of my pixhawk-type FC’s I have ended up figuring out which is the best IMU and running with a single EKF with much better and much more predictable results. And like @wicked1 I would say that bad hardware is the norm here. None of my FC’s have all sensors operating as they were intended.
Hmmmm… that’s interesting. I didn’t catch that. The MPU6000 has the gyro and accel on the same silicon die with an onboard DMP. How is that even possible?
The accelerometers agree in the same direction. But the gyro on #2 is indeed rotated 90 degrees:
There is, but after Randy pointed out the x vs y on the 1 and #2 gyros it sure looks to me like Invensense built it wrong. Those MPU6000’s were used on just about every board that flies for several years. And sometimes they are noted for drift and bias problems. But I’ve never seen one with the gyro rotated 90 degrees in the die!
The board is still fully flyable. Just don’t use the second IMU.
In Mission Planner’s popup for the EK2_IMU_MASK you should be able to select which IMU’s you want to use. If you want to run just one EKF process select just either 1 or 3. If you want to run dual EKF processes select IMU’s 1 and 3 (the IMU numbering in MP’s popup box may start from IMU0 which is the first IMU, can’t remember).
Even though the EKF will only use those IMU’s the output from the bad one is still logged. So I don’t know if that will get rid of the message about inconsistent gyro’s. I think it will - you will have to try it.
@KbGo I am setting a helicopter to do some Acro code testing so took a screenshot of this for you. I guess MP names the IMU’s so it is pretty simple.
I do not have a board with a bad gyro to test to see if still get the inconsistent gyro message if there is a bad gyro on the board, even though it is not used. Just have to try that and see what it does.
