Hello,
We have a big issue with lasted firmware arducopter 3.6.6.
After some flight (5) we make a last mission and all work well.
But at the last mission loaded, copter do a FLIP on the roll at 180 ° and the top now look the ground, and motor speed up at 100 %. The drone fall than 10 meter…
Nothing in the logs, no error or something can explain why the machine do a flip.
We used Pixhawk cube 2.1, rangefinder, hex GPS 1 like our other machins but flight in 3.6.1 or 3.5.7.
No error or anomaly on RCIN or RCOUT.
No change mode
No EKF Error
It’s like the flip is requested by the firmware…
When the drone done the flip it control there postition and continue to stabilize like it normal but not in the good way !!!
I think it’s an issue in last firmware.
I don’t know where i must put this issue, and if i must upload our logs etc.
thanks for helps.
Thanks for the report, this is a very unusual log. It appears that the 2nd EKF (the backup) flipped over at some point while the vehicle was disarmed. The primary EKF was OK which is why the vehicle flew OK at first. Just before the crash, the EKF selector saw that the height estimate innovations were very high for the primary EKF so it switched to the backup. This was a poor choice by the selector because the attitude estimate was very bad.
Do you remember doing anything while the vehicle was disarmed? Perhaps a compass calibration? Any chance you have tlogs for this period?
I see the ARMING_CHECK is set to not check the compasses. Disabling arming checks is not a great idea because it does disable checks that might stop the vehicle from taking off in an unsafe state. If the vehicle’s is sending the “inconsistent compasses” messages it is better to set COMPASS_USE2, COMPASS_USE3 = 0 as mentioned on the wiki.
There are two things we can do:
add a pre-arm check that the EKF attitude estimates are within some margin (like 10degrees) (issue raised here)
investigate why the EKF-chooser made a bad choice
Thanks again for the report. I think we can make AP catch this situation so other vehicles are saved. sorry it was too late for your vehicle.
Another couple of things I’ve noticed from the logs that may not be related:
the RTL_ALT parameter is being updated constantly (but it’s not being changed)
loop performance is not great with about 10% of loops running longer than they should. It might be good to switch to the ChibiOS build which runs faster.
@rmackay9 It flips precisely at arming. And I think it is caused by using a DJI-style arming scheme and pulling both sticks down and to the center, moving the ghost, and causing the EKF to flip over.I had this happen before (I think in 3.5) when doing cyclic checks with a helicopter but didn’t crash due to having enough altitude to recover. If you graph the mag-y I think you’ll see they all agree and they will show exactly what happened - at least it did with mine. It is not a mag problem.
The solution I came up with, and have used for a very long time, is to disable the extra EKF’s with the EK2_IMU_MASK setting. And do not use stick arming. I know that is not recommended or popular, but I have never had a problem with one causing a radical attitude upset with EKF “lane switching” since.
Thanks for that, the pilot does appear to have all the sticks at the extremes as they arm instead of just throttle low and yaw all the way to the right but I’m not aware of any issue that would be caused by this. I know of these additional “gestures”:
compass calibration : yaw to right, throttle to full
auto trim : yaw to right, throttle to minimum, hold for 15 seconds
Autotrim is the most likely because it affects attitude but it has a limit of just 10 degrees and it should apply to all EKFs not just one.
I’ve created this PR which still needs to go through peer review but I hope will stop this problem from happening in the future:
always run pre-arm check that forces user to reboot after performing a compass calibration (previously this was disabled when the ARMING_CHECK bit for compasses was disabled)
check the attitudes of DCM, EKF2 and/or EKF3 are consistent within 5 degrees of roll and pitch, 15degrees of yaw
Thank you Very much for reply.
Ok now i understand why drone was flip.
But it s sur the pixhawk was reboot after calibration.
We made some flight with drone without problem before and we reboot drone After each.
For alt_rtl it was just a test of my custom ground Station.
I will change it.
I think, i take of on a Bad electeomagnetique environnement . Maybe cause the problème.
It s why we disable compass in pre arm check. But we done it since 2 year, ans i never had problem before.
Thank you a lot, i will do some test today and i come back to report that.
I will lookup log before the last flight of my drone.
OK, i begin to understand the problem.
I think we have a problem with on off compass and EKF2 make an error with roll inversion.
But we used a rangefinder connected with serial port. Sometine there are errors on read and in case of error the value is 130 meters.
I don’t understand why EKF1 used this value, because the read max is 120 meters.
When the rangefinder have an error and indicate 130 meters, the ekf do an error and switch the primary.
In my log we can see that we normaly use EKF2. After the first flight and first landing, the drone was stay on ground in a perturbate environnement magnetic. we have some GPS glitch and compass errors. The EKF1 stay correct, but EKF2 roll drift and flip to 180 degree. (i don’t understand why when it’s disarmed ekf don’t saw with GYRO + ACC that drone are not realy flip and why it don’t do an critical error)
So because there are GPS drift, ekf primary switch to ekf1. I takeoff for a second flight (without reboot my drone) without problem, but ekf2 stay in error with inverted roll . Drone done the mission, and we have a noise on serial communication on the lidar just for some milliseconds(lidar read 130 meters)… EKF1 write an error and switch primary and come back to EKF2 with inverted roll. Drone flip and crash.
Today I changed some params on my config for rangefinder, but i don’t understand why value 130 m is used on innovations EKF while the value max of range finder is 120 m.
I’m sorry my english is not good, i hope you understand my feedback.
Thanks a lot.
Thank for your reply, the flip was made in the air with an altitude 10 meters, not just after arming.
After some seconds on flight and after a mission .
I don’t know why ek2 roll was drifting but the problem is not with rc.
Thank you
@rmackay9 I refer to the desired attitude as the “ghost”. With helicopters we used to be able to move the cyclic to the limit and it would rotate the “ghost” and cause the swashplate to flip back and forth as the “ghost” went inverted past 180 deg roll. It was noticeable in Acro, and even though it wasn’t a real problem pilots found it disconcerting to have the swashplate doing that. We are able to see it in heli because the servo outputs are active in unarmed state. So when pilots did cyclic checks they would see this, either armed or unarmed.
Bill put a patch in to zero the attitude on heli’s until, IIRC, throttle hold is released. I don’t know if this is related to the patch we put in for heli. But it sure looks like the same thing, as my log tool shows the NKF6 Roll going from 3 deg pos to 177 negative exactly at arming.
I don’t see any anamolies in the mag y outputs.
Maybe @bnsgeyer remembers the details of that patch, as we assumed the problem only applied to helicopters.
It didn’t drift. From the log at 362 seconds after the first disarm the value of the roll was 3.05. It flipped at the next arming event to -177 - exactly 180 degree flip. The way the software draws the graph it appears as a gradual drift to the inverted value. But it’s not. If I remove all the errors and other messages from the graphing you can see how the software draws the linear interpolation from the disarm event to the next arming event when it instantaneously flipped by 180 degrees, simultaneous with the cyclic input on the RC.
Ah, the EK2_ALT_SOURCE = 1 (RangeFinder). This should be returned to “0” (for barometer). Setting this to “1” (RangeFinder) is almost never a good idea and it’s not necessary for terrain following in manual or autonomous modes.
I’ve modified the EKF wiki page to make it more obvious that this parameter should not be set to “1”. As a a side note, I feel like I’m constantly battling to stop people from changing this parameter… I may add a pre-arm check to try and stop people from changing it if they have a GPS active.
It does sound like we may have a bug in the EKF’s integration of the range finder so I’ve created an issue. I’m afraid we are a little low on EKF developer time so I can’t promise when we will look into this. I hope we can deal with this issue by returning EK2_ALT_SOURCE = 0.
Txs for the info. It doesn’t really make sense though that moving the sticks around (which controls the target attitude) would affect the estimated attitude. The two are completely separate …
Anyway, I think we’ve determined where the issues are for this crash report and hopefully we can resolve them with some pre-arm checks and perhaps some work on the EKF range finder integration.
Yes, i see it. I change ALT_SOURCE and i have no switch EKF now.
I understand about the switch, but not for the roll flip.
@ChrisOlson
It didn’t drift. From the log at 362 seconds after the first disarm the value of the roll was 3.05. It flipped at the next arming event to -177 - exactly 180 degree flip
Yes, it’s an error of interpretting the log. No drift, it’s flip when arming. it’s strange.
