I have a consistent problem with Position Hold, if I do a compass & accelerometer calibration, the quad always flies absolutely perfect in position hold for an entire battery, no compass variance or other warnings during the entire flight, and I can cycle through various flight modes with no problems.
Always on the next flight without calibrating compass or accelerometer, the problems start, it always flies with a toilet bowl oscillation in postion hold(generally increasing in severity with time), repeated compass variance warnings, EKF2 IMU ground mag anomaly warnings, and repeated switches between compass 1 and 2, and switches between IMU 0 and 1. Nearly had a serious crash just after it switched IMUs near the end of the attached flight log, it twitched abruptly, then rotated around the Z axis in a wild toilet bowl rotation, managed to get it on the ground with no damage - barely.
Other than recalibrating compass and accelerometer before every flight, is there any way to resolve this? After comparing logs of flight with calibration vs following flight, INS_gyro offsets are significantly different for both IMUs, but all other sensor offsets I could find remained unchanged. I do have the “learn offsets automatically” checked under the compass settings section(default I believe), does that be alter gyro offsets?
As no-one has replied I thought I would chime in and let you know there are similar issues being raised in other questions.
There is definitely something going on in the EKF implementation so I guess for now it’s just a matter of us continuing to give the Dev’s our log files and hope they sort it out soon.
I had a quick look at your log but to my untrained eye I could not see anything definitive.
All this new EKF stuff has got me so far out of the loop I am not much use there.
Thanks for the reply Mike, and yes I do think that’s the case, there’s a lot of new EKF related stuff in 3.4, and its very complicated to say the least. I am going to try swapping out an original pixhawk on the quad to see if it could be hardware related being I was running a pixhawk clone on the quad so far. Also, I do see there are some new filter parameters for autotune, and they recommend if you’re running props 13" or larger(I’m running 15"), you need to change the filter frequency to 10Hz versus the default 5Hz, that may help a lot, definitely can’t hurt at this point.
I believe the default was 20hz for roll/pitch and it should be lowered down to 10hz. The 5hz default is for yaw and the latest autotune had tuned it automatically for me down to something around 1.5hz so I believe it is not necessary to change the yaw filter.
Anyway , I have two compasses, one px4 internal and the other is an external 3dr gps, What I did to solve the issue and get rid of all compass errors:
reset px4 back to the default
uploaded fw.
updated all the parameters previously saved , except all compass settings
did an onboard mag calibration wireless ( using 3dr telemetry ) with all my electronics connected (gopro gimbal etc )
did compassmot calibration wireless ( using3dr telemetry ) - everything powered on again
Problem is gone !
I noticed that the onboard mag calibration procedure sets all compass soft iron matrix parameters, which I couldn’t find any documentation besides the code itself. I’m not sure if MP’s live compass routine should also set those values, but it didn’t happened with me.
To start onboard mag cal, just click on the button and start slowly doing all movements you can imagine and wait for the progress bars to be filled. In the end it should be successful.
A note on gyro: one of my drones has a compass very close to the gimbal. I had to power it off and tape the gyro to prevent motor movement. When they were free, the calibration ends in error. The other one doesn’t had this error because all compasses were away from gimbal.
Other thing important , in one to ten times I try to arm i got the compass error. I I grab the drone and to a full 360 turn with it , the ready beep cames and it becomes ready to arm , always!. That could be the learn offsets routine doing its job ( just guessing -please correct me someone ) .
Thanks @tabascoz,
I just completed a yaw auto tune and the changes are very similar to yours, especially in the Yaw Filter Freq.
The PID was increased more than 10 fold, a lot more than I would have ventured manually.
The yaw variances are no longer there and it Loiters very well now.
Ok, sounds like the onboard mag calibration may help a lot, will definitely give it a try! I did the compass-motor calibration previously, but didn’t seem to have any impact on the problem, so it must be something else.
I had similar issues in the last month with a brand new copter with a Pixhawk clone, Mauch main Ubec and Mauch current sensor.
Two differences:
the problem arises randomly , three or four batteries perfect and then it appears again . Mainly with compass calibration and EKF compass so also Loiter performance but also more rarely other errors like GPS horrizontal errors.
At the beginning it happens almost every time I flew, then after several compass calibrations and using only external compass the problem seems solved and arises rarely until last time.
the last time it happens it was incredible, it flew perfectly in the morning and two hours after it was a mess, not only the mag and EKF errors but a completely different behavior with all kind of errors as, vibrations that went from 1.2 to 5 , Des roll and roll totally different and the copter act as it was drunk, throttle that change from 1500 to 1000 even if radio and receiver were perfect and I did not touch the radio stick , briefly the copter was impossible to control and need a exorcist.
I will post the logs , quite interesting …
I spent three hours to find out what the hell was going on, turning on and off several times the copter but the problem remains, I just could understand it was not a mechanical Esc/motors problem.
The day after I switch the Pixhawk clone with a original Pixhawk on a twin copter , both flew perfectly without having done any configuration change.
My conclusion is that the problem was an interaction between the Mauch power supply and current sensor and the Pixhawk clone that occurs randomly.
To be honest and not want to accuse Mauch products , three other switching Ubec of the same type as Mauch Ubec, one for navigation leds one for video tx and one for main supply backup were present.
I will check with an oscilloscope the noise of the power supply system .
It was a very sneaky problem that appears as a calibration / software problem but in fact was a hardware problem.
Never saw anything like that in 8 years of work with multicopters.
P.s. Now the power supply and current sensor is an AirbotPower from Airbotservices, a great product.
That is very strange behavior indeed, I’m currently running an HK 10S power module(10S clone of 3DR power module). I also wondered if the problems I’m experiencing are all clone related, so put an original pixhawk on it, but only have the HK clone 10S power modules. Will test tomorrow to see how it goes. I also tried running with external compass only, but that didn’t help at all in my case, so its definitely something else…
I haven’t looked at the log yet but there’s certainly an issue in AC3.4-rc5 (and earlier) where it can move suddenly when the EKF “changes lanes”. We’ve got a PR from Paul which is 1/2 the final fix. https://github.com/ArduPilot/ardupilot/pull/4903
To explain in more detail, one of the key changes in AC3.4 is that we have multiple copies of the EKF running. In AC3.3.3 we had a single EKF that used multiple IMUs. In AC3.4 we have a single EKF for each IMU and we pick the one that reports it has the best consistency. These multiple copies have been called “lanes” it seems. So when we switch EKFs the estimated attitude, position etc can jump suddenly and this can lead to a jump in position in Loiter/PosHold. We can handle the shift though elegantly by telling the position controllers whenever the position has jumped. So Paul’s PR linked above can tell the controller… the last thing we need to do is add the handling in the controller. That’ll go into -rc6.
Ok I finally figured out the cause of my position hold issue, its battery related believe it or not. With this quad rotor, I’m running two custom-built Li-Ion battery packs not LiPo. I thought I had them mounted the same, but it turns out they were not, luckily, the builder sent some photos as they were being built before he added the heat shrink covers, this way I could figure out how the second battery needed to be adjusted so both batteries are oriented exactly the same on the aircraft.
Once I got them both mounted exactly the same, it flies perfect now with both batteries, no compass variance warnings and position hold works like it should. In this case the battery orientation matters a lot being I’m pulling a lot of amps (35-45), and the batteries obviously generate very large irregular shaped B fields.
Would you mind explaining this a little more? Was it the location of the batteries relative to the compass? As in were they not far enough away from the compass? Or was it an orientation thing relative to the compass?
I am experiencing the same issues on my coax copter and wondering how battery location and orientation may be factoring in and how to mitigate interference.
Joe, it took me a long time to figure out, about 2 months or so, but my problem was related to the high amp draw of my batteries(35-45amp), and I was using two separate Li-ION battery packs, the cell orientation and contacts of the soldered tabs, was slightly different between the two packs. I did a compass-motor calibration with one of the packs, and the other pack was significantly different( I assume due to the size or shape of EMI field), and shortly after takeoff(with only one of the battery packs), I would always get several compass related warnings on takeoff, and position hold would not work correctly, it would always oscillate in a toilet bowel fashion and get progressively worse. Only after changing on the order on which pack I started with (Pack1 or Pack2), I finally realized this problem was battery pack related. With very high amp draw airframes, I believe it is absolutely essential you perform a compass-motor calibration only after you have calibrated your compass. Also, if you use multiple battery packs of the same size and specs, they must be 100% identical, otherwise the compass-motor calibration will be invalid.
Hope this helps, let me know if you have any other questions.
Forgot to mention, this problem was not battery location related( I don’t think I could’ve re-positioned either the battery or compass to make a difference), it was only related to differences in the EMI field and shape relative to both battery packs used, meaning only one battery could be used successfully if that same battery was used for the compass-motor calibration, the other battery pack had significantly differently contacts & cell orientation, so its EMI signature was vastly different, to the point that the compass-motor calibration would only work for one of the battery packs.
I have a single copter running about 100amps, I have 4 identical batteries. Would shielding the batteries or compass with EMI shield / foil have a similar effect to compass-motor calibration?
Hi Roly, aren’t you using a current version of Arducopter?
You can do the compass/motor calibration and disable the onboard compass if there’s one in your GPS unit…
Are the batteries LiPo or LiIon? LiIon batteries always need to go in the same orientation. Different packs would need identical construction.
Mu Metal will shield against magnetic interference if there’s no other way. You would want to put some shielding around the batteries, not the compass obviously. A shielded compass is not a compass
I’m using a Here3 GPS, I suppose I can try using the Cube’s internal compass for a change but the Cube is actually closer to the battery and ESC than the GPS, so I would expect it to get more interference.
Wouldn’t shielding the GPS puck on the underside help it and the compass?
I’m using LiPo batteries and a current version of ArduPilot. I just ordered a current sensor as the stock one doesn’t support 100A
