I recently upgraded my system to Copter 3.5.3, I flew about four short flights and everything worked well (AltHold). All flights were done inside in a warehouse. The next day I wanted to calibrate the current sensor and went to fly it again. This time, about 1:30 into the flight, the altitude ran away and it smashed into the ceiling (about 20ft up) then fell back to the ground. I haven’t been able to figure out the exact cause. I’m not sure if it was an issue with 3.5.3, it never had any problems like this before. What is weird is the Baro shows the change in altitude and the EKF for the Pos(Vert) pegged into the red (you can replay the .tlog in MP). It does not appear to be a vibration issue either. The climb was very constant and not full throttle. it looked as if someone told it to climb normally. Unfortunately I did not have a Stabilize mode setup, which I believe would have allowed me to save it. Hopefully someone can help find the problem.
Thank you for taking a look!
It actually crashed into the ceiling of the warehouse. I was pulling all the way down on the throttle trying to stop the climb, with no luck. It kept ascending until it struck the ceiling and fell back down. if you graph the BARO.alt against CTUN.DSAlt you can see where the two separate (these should be close to matching?)
FWIW, I once had a low battery trigger a RTL, which made it try to climb to RTL height, so it could land, but that height was higher than my celling. But I didn’t see a FS message in your log. Just mentioning this as you said you were working on battery sensor cals. I did much the same trying throttle etc., before I hit Stab and was able to cut throttle. Sorry, but No idea on the divergence you have in your graph.
Joe.
Hey CrazyOne, I have been looking at your log. I know what happened, but not why exactly. Looks like you were hovering and there was a slight blip in the z accels at ~ 21.5 and then it was climbing and you lowered the throttle stick as is climbed to the roof. Your action probably slowed the climb.
Barometer altitude shows the climb after the anomaly.
The altitude from the ekf didn’t match the baro. It went down at a constant rate. As the desired altitude went down from the stick being down in alt hold, the " actual alt " followed it into the roof :(. It would have went up faster if not.
Why this happened is beyond me, but it seems to be an important find. Vibe were good, no clipping, motor outputs good.
the EKF 1 z position and the EKF 2 z pos diverge at the same time.
your copter followed EKF 1 because there were no errors to cause a switch over. I don’t see any thing in the raw sensor lines, just in the EKF pos down value.
I will try and get the right people to look at this. Paul R?
Thank you for taking a look. Ive seen lots of crashes and typically there is a logical solution after reviewing the logs. This one has me perplexed because everything “appeared” fine. The only time I have seen a similar issues was due to bad vibrations. IMU1.AccZ & IMU2.AccZ do appear to look a bit strange (at least to me) while they both seamed to be within error range, IMU1 had virtually no vibrations but IMU2 saw much more. On past frames the logs typically show both IMU’s having roughly the same variations. I know the PixHawk has two different IMU sensors to combat aliasing situations and it depends on the vibration frequency. was there a “perfect storm” that caused the PixHawk to get confused on this flight?
The Image below is from the crash. The two IMUs appear completely different.
This image is from a different PixHawk and similar frame. It appears to look more “normal” in regards to both IMU’s showing roughly the same vibrations.
Yes, your IMU2 is quite noisy, but that won’t affect your first EKF estimates. It could mean that it’s not worth running the second EKF, though (EK2_IMU_MASK).
I’m not sure what went wrong exactly, but here are my notes:
Without GPS or a rangefinder, EKF altitude is fusing only accelerometer and barometer (I checked your params to make sure it wasn’t attempting to use other sensors). This suggests that the accelerometer might be the source of the error.
Except for initial takeoff, the climb rate and desired climb rate is negative for the entire flight, around -20 to -30 cm/s while maintaining altitude. This is anomalous; it should be near 0 while hovering.
Climb rate is strongly dependent on IMU. This could mean that it needs to be recalibrated or didn’t initialize correctly, as it appears to think that it has a constant downward velocity. However, the EKF is supposed to use other sensors (e.g. barometer) to learn these constant biases.
EKF IMU1 Z bias increases steadily throughout the flight, but it doesn’t seem to have any correlation to other measurements.
It seems that your EKF is ignoring your barometer for some reason. You can increase barometer trust (and reduce accel trust) by lowering EK2_ALT_M_NSE. This shouldn’t be necessary as the default values should work and both your sensors seem well behaved, but it’s an option if we don’t find the cause. Might be worth trying.
What I would do next is look at previous logs from this copter and see if there are similar issues, such as climb rates that don’t match altitude changes, or EKF alt and baro alt divergences.
Anubis,
I added a new folder to the above link. It contains the three previous flights that were done after updating to 3.5.3
Also,
Today I did some more testing. I flew another system similar to the one that crashed. I loaded 3.5.3 and made sure to have a Stabilize mode . AltHold did very, I had the Pos(Vert) up on MP during the flight and it sat in the middle of the green going up and down slightly.
I didn’t like the look of that so I decided to rollback to 3.4.6. Sure enough it flew very well. The Pos(Vert) barely even registered. It was typically nothing. The vibration Z showed the same for both 3.5.3 and 3.4.6 but was well within the green. I am not sure how the AltHold structure was changed between 3.5 and 3.4 but there definitely seems to be a difference. I always try to keep my systems up to date, but maybe ill sit this one out…
These flights were done with a different PixHawk, but similar frame.
Here, when the drone is resting on the ground, the EKF altitude decreases, even though this disagrees with the baro. Log 38 has this also, but in the opposite sense: a positive climb rate while the drone is resting on the ground.
I’m not certain where this error is coming from; the few examples I’ve seen were GPS-related. Also, I don’t have much experience with 3.5 yet so I’m not sure what could explain the difference in flight behavior you’re seeing.
My pixhawk firmware version is 3.5.2. Quadcopter was in stabilized mode initally and then loiter mode that made the quadcopter rise suddenly. For firmware 3.4.6, I never encountered this issue. I think this situation is very similar to yours.
Guys, this issue has been fixed by Paul with this PR with a total of 38 commits:
The PR has been merged into master, but is not yet in the current stable release. Hopefully when Randy gets a chance to get it into 3.5-Backports the fix will come out in Copter 3.5.4. There is a new param, EK2_OGN_HGT_MASK, set to zero will force use of the baro for the height origin.
I fly helicopters and am currently flying the patched code built on Copter 3.5.3. Information on the issue is in this thread in the Trad Heli section of the forum:
The patched code for Copter 3.5.3 is in my ArduHeli master branch on GitHub, but unfortunately I have only done builds for helicopters at this point.
Thank you everyone for the help!
I feel much better knowing it is not something I did. It always sucks watching your new build go out of control and crash. Its even worse when it could have been avoided. I have since been running 3.4.6 and have seen no similar issues. I will keep with 3.4.6 and maybe give 3.5.4 a try when it gets released. Hopefully threads like these help others from making the same mistake and having issues.
I believe the problem has been noted in 3.4 as well. But there were other changes to 3.5 that may have made it more manifested.
If you would want to install the latest beta (3.6-dev) from ardupilot master the EKF updates are in there. I simply pulled them out of ardupilot master and merged with my ArduHeli master branch so I could fly the latest EKF code on 3.5.3 in my helicopters. I found the problem to be quite pronounced at higher flight speeds >20 m/s, with the EKF altitude estimator off by 17 meters on one flight. The problem has not re-occurred flying Paul’s updates.
. AltHold did very, I had the Pos(Vert) up on MP during the flight and it sat in the middle of the green going up and down slightly.