Very weird problem acceleration

Right, that was what I was going to ask - what is the weather like? The more expensive flight controllers have IMU heaters to avoid calibration divergance. If you are flying in very cold weather I can imagine things shifting. Have you tried insulating the IMU chip?

The weather is coldā€¦ But I think itā€™s more than just that. I fly in the cold every dayā€¦ The issue is present on other copters in the summer (though not this bad). And, it does it insideā€¦
It is a new kakute mini, and I have two of them. They are both like this. All my omnibus nanoā€™s do this too, to varying degrees. (Iā€™ve flow 3 omnibus nanos regularly for years, and now 2 kakute miniā€™s all winter)
Itā€™s only when I do not have GPS. As soon as I see the message about EKF using GPS, itā€™s fine. But I often donā€™t have GPS on these little microcoptersā€¦ Flying in the woods under the tree canopy, flying indoors, etc.
The issue is not present on any of my 250mm and larger copters.
At least, not that Iā€™m aware ofā€¦ I guess I always fly those where I have GPS reception.

Things me and a couple other people who have posted about this issue have thought as the cause have been electrical interference, either needing better filtering, or maybe the 4in1 ESC being mounted directly under the FC causing problems. (Iā€™ve tried caps. Canā€™t do anything about the ESC)
Someone was sure it was related to the load on the processor. (Iā€™m fairly sure thatā€™s not my issue)
I was quite confident vibrations were the cause, but not so much anymore. (Fixing vibration issues on my last kakute mini definitely made it better, but I donā€™t think I can improve them on this new copter)
Temperature, like you guys are suggestingā€¦ But I donā€™t think thatā€™s itā€¦ all these small FCā€™s build up the same gyro bias in any weather. I will try insulating the IMU, though.
Since every one of my small copters does this, Iā€™m surprised more people havenā€™t noticed the issueā€¦ But maybe most people wait for GPS before flying. I know at least 4 or 5 people have posted about the issue, all using sub 250mm copters. But weā€™ve all had different theories to the cause. Iā€™m not sure if the other people solved it or just moved on to other projects.

I just looked at logs from all 5 of these small boards (3 omnibus, 2 kakute), and ONE does not have the ever increasing bias on the X axis. Iā€™m not sure what to think about this at the moment.

-Edit, no, theyā€™re all about the same. I looked at a lot more logs, and apparently they all occasionally have a flight where it doesnā€™t increase the bias much.

So a few things to consider.

  • My guess would be that the leans while no GPS is because the EKF cannot compensate for the X-axis offset without the additional data that the GPS provides.
  • Accels are not calibrated on startup. Have you tried re-calibrating in the field when the temperature has stabilised?
  • It would be good to see a log with the raw accel values for comparison. I suspect this will look the same, but would also be good for comparison. Will fill up your logs quickly so beware.
  • You are flying in stabilize right? Can you reproduce this with the GPS disabled and in a stable hover? That would be interesting for comparison.

Iā€™m struggling to get a logā€¦ By the time I land theyā€™re 16MB, and when I try to graph things, itā€™s all just a straight line. I think it basically starts overwriting itself and gets corrupt. Iā€™m heading to work but will try again later or tomorrow.

What seemed odd. I took my first flight by quickly arming and flying, w/out any warm up period. It wasnā€™t leaning badā€¦ A bit, but I could deal w/ it. I landed and accidentally saved the wrong log file, then cleared the logs, so no data. Then my next flight it tilted VERY badly. But, the copter had been inside and warming up for a while. Temp is about 40F outside. I had to land because I couldnā€™t really control it. I just pulled the battery and plugged it back in, took another flight, and the leaning was not too bad again. So that could be pointing to your temperature ideaā€¦ But I really donā€™t think thatā€™s it. I fly all of these in a huge range of temperatures. I was flying in below zero (F) weather a few weeks ago w/ my omnibus copter, after an inside warm up period, and didnā€™t have any problems.

One thing I noticed about the leaningā€¦ It started leaning to the left and got to about 45 degrees. Then I yawed around so forward was pointing to what was the left side, and as I yawed, the copter tilted up. So the left roll became a back pitch. Then it leveled itself and started tilting left again. The direction of the error basically was directional based on the earth, and not based on the copterā€¦ or somethingā€¦ I was able to repeat that a few timesā€¦ (My other copters are probably the same, but the amount they lean is not nearly so bad, and I donā€™t really notice the several degrees of pitch being off after I yaw around)

Well anyway, off to work. Iā€™ll try logging raw again while doing a stable hover when I get some time.

A log is key :slight_smile: and sounds like a little detective work might give us a useable clue - it feels like we are close to some better insight.

Make sure you are using the latest release candidate of copter as it includes my logging fixes which should clear up the overwriting issue you are seeing

I am using rc4ā€¦ Still logging issues. I can see legit data in the table for gyr, acc, etc. But both MP and APM Planner are only graphing straight lines.
I guess Iā€™ll upload itā€¦

The log looks valid. I think the issues you are seeing are not log related, but perhaps scaling related - so the incredibly large peaks are hiding the actual data on the ā€œflatā€ line.

The incredibly large peaks are worrying, that should not be happening, that points to some kind of sensor overflow I think.

@tridge any thoughts on the raw sensor data in this log?

Not sure if itā€™s related or an issue, but I often donā€™t get ā€œtilt alignment completeā€ and ā€œgps saving configā€ messages until after I land, after about 6 minutes.

Iā€™ll get another log later to see if those same big peaks are in it.

There is also somewhat of a correlation between some of the peaks and your battery voltage:

Insulating the IMU seems to have helped the bias growingā€¦ But it still leans the same.
In fact, I had GPS lock, and it still tilted! This is the only copter thatā€™s ever done that to me.

I am actually starting to wonder about a bad FC againā€¦ This is the one I originally used when I started the ā€œKakute mini problems (or just problems w/ my build)ā€ thread. We got side-tracked by all the vibration issues, and I did replace this FC w/ a new oneā€¦ and at the time I just focused on the vibrations being the same on the new one as this one. But maybe the leaning has been better because of the new FC. URG, I donā€™t knowā€¦ I tried to get Holybro to RMA it a month or two ago when I first had the issue, and they flat out told me Arducopter isnā€™t supported. Show them a screen shot of the IMU test in betaflight failing, or else they wonā€™t RMA it. This issue is too subtle for the IMU test in betaflight to find a problem.

I couldnt find all the parameters (or much else) in you log files. I was looking for:

INS_GYR_CAL 1

And what value do you have for EK2_GPS_CHECK ?
Set this:

LOG_BITMASK 141310

And then get us another log to look at

The log file is corrupt
@wicked1 did you pull the log straight from the sd card or over mavlink?

It will be over mavlink as that board does not have an SD card

Out of curiosity - what makes you think it is corrupt? It will start abruptly because itā€™s a wrapped 16Mb FRAM log and APM Planner wonā€™t like it because there are ISBH and ISBD messages in there, but I wasnā€™t seeing any other problems?

INS_GYR_CAL is 1
EK2_GPS_CHECK is 31 (allows arming while moving, which I need)

That is what my log bitmask is atā€¦ My last one had IMU disabled because I was collecting raw_imu, but the previous one should have been 141310, and I will go take a flight now w/ 141310 and post the file in a while.

Well, itā€™s raining/snowing. I did get a hover inside with the tilting issue, which might actually be more helpful than a log of me flying around outside.
The log has two hoversā€¦ The first one wasnā€™t badā€¦ I am having to slightly compensate for the tilt. Then I landed and the copter sat there for a while. Then the second flight was more typicalā€¦ The way I have it trimmed is with some extra right trimā€¦ So, I take off and am having to pull it left for a few seconds. Then the issue develops and it levels off and starts pulling left, so I am compensating by giving it right roll.

Iā€™ve just been randomly trying thingsā€¦ Tried various gyro and accel filter settings, taking gyro all the way back down to 20. Tried various settings for the ek2_gbias_pnse parameter. Tried GSCL_P_NSE, GYRO_P_NSE, put the loop rate back to 400, tried oneshot instead of dshot. Nothing made a difference in the leaning.

It starts leaning just seconds after I take off. Then after I land, Iā€™m sitting level on the ground, but the HUD in MP shows me at an angle. The HUD starts leveling itself, and after sitting unpowered for several seconds, the HUD in MP shows me level.

It seems like something in the logs would have to show this difference from reality, but I canā€™t find itā€¦ Other than looking at my roll RC input, and seeing me manually compensating for it. And I did notice AHR2 roll and ATT roll are different from each other. The worse flight seems to be more different.

Thanks again for everyoneā€™s help. Iā€™m close to giving in and purchasing another new FC to see if it makes a differenceā€¦ (anyone have any contacts at Hollybro to get a development sample? This is getting expensive :slight_smile: )
Iā€™ve mentioned before, all my copters w/ these small boards do this, but to different degrees. A couple years ago when I was experimenting w/ the omnibus, I basically had 1 thatā€™s so bad I canā€™t use it, and three that were OK, but some worse than others. Now itā€™s looking like a similar thing w/ the kakute. This one was badā€¦ I replaced it and the next one I was able to get working satisfactorily. (I assumed because of vibration mitigations) But now Iā€™m trying to use the original one again and can not get it working, with even lower vibrations.

So I think you can see what you are talking about in this graph:

The AHR2.Roll is the inertial roll only (I think) whereas the ATT.Roll is the EKFā€™s idea of roll. You can see that they diverge quite dramatically in the second flight. You can also see that the innovations for GPS velocity measurements in the north direction vary dramatically as well (https://ardupilot.org/dev/docs/extended-kalman-filter.html) the docs say these should be small and centered around zero. Yours are anything but - and that seems to be because the GPS started working half-way through the second flight.

Can you try this - physically disconnect the GPS, re-calibrate the accels on site (you still have a large accel bias, but it is constant) and then see if you can reproduce the problem?

I did calibrate accelā€™s this morning before my tests. And, I did it wellā€¦ Attached the copter to the inside of a box, and had it level and still on each side as I calibrated.
GPS is likely because Iā€™m indoors w/ a quite bad signal. The issue is definitely present w/ no GPS. I did for a couple flights this morning turn off the serial port (put it to -1) so it wouldnā€™t talk to the GPS, and those flights still had the issue.
I can get a log w/out the GPS, thoughā€¦ I unfortunately overwrote the logs from when I did it this morning. I D/L them, then cleared them. Then messed around w/ the copter for a while (tested vibrations per motor) and downloaded those logs. Since there was no GPS, they had the default file names, and it just overwrote my original logs.
It would be really nice if MP had a save dialog for the log downloaderā€¦ Would be helpful for people w/ multiple copters, and apparently for when thereā€™s no GPS data to rename the logs.

Ok, from your memory of the second flight what were the leaning characteristics? You can see the two crossover midway.

FWIW I donā€™t think you get dated logs with FRAM even if the GPS is working, I looked at trying to make this work and it wasnā€™t that easy - I agree save as would be nice!