Loiter mode resulting in crash

So this is a plea for help… I Have flown drones for years With betaflight firmware. But when it comes to Ardupilot and its diagnostic tools, It’s like I’m relearning the hobby all over again.

So today was the first day in about six weeks that we had a decent day of nice temperatures and no wind to do a little bit of tuning and flying. I started with trying to set my filters and I believe I have that somewhat squared away. However, I would encourage anyone to Double check me.

But the main reason for this post is I was in Loiter Doing some simple Maneuvers and watching the GPS hold. And then the quad started acting really weird. Flipping back and forth Not violently, but slowly, gradually getting worse. and eventually ending into a crash. It happened so fast I can’t really Recollect exactly what the quadcopter was doing, just that it was doing things it wasn’t supposed to be doing. I did disarmed on the trip down. I landed in some Plum bush’s so thankfully only a busted prop.

I suspect there are issues with the GPS and or compass, but my lack of knowledge with these diagnostic tools have made it very difficult for me to try to figure out what the problem is.

If someone can take a few minutes to review this log, I would greatly appreciate it and point me in a direction that could help solve this issue.

2024-01-28 17-31-52.bin

Hi Wes,
The copter appeared to be going well before losing attitude control.
The odd thing is it does NOT look like a motor, prop or ESC issue, since none of the motors are commanded to maximum.
GPS seems OK - so far…
IMU seems OK, vibrations acceptable.
Battery voltage seems OK too.

Right around the time of the instability the GPS position and IMU-calculated position start to diverge but I’m not exactly sure why yet.
There are messages about “EKF variance”, “EKF bad” and “Main Loop slow”, and you only have one IMU to rely on too.
The CPU load does increase over time.

So I would work on reducing the CPU load:

FFT_ENABLE,0
SRTL_POINTS,0

I would measure the supply voltage to the GPS unit and ensure it’s right up to exactly what it should be, 5.0 to 5.2 volts.
Also reduce the number of constellations to improve the update rate.
GPS_GNSS_MODE,7
or 67, whichever gives you the lowest most reliable HDOP.
…and a couple of general improvements:

BATT_ARM_VOLT,14.70  // maybe you lowed this to allow arming more often
INS_ACCEL_FILTER,10

These are the Harmonic Notch filter settings you should use to make use of the ESC telemetry instead of FFT. The in-flight FFT is not picking up on the correct noise peaks, so it can safely be disabled, as stated earlier.

for this outcome, other peaks you can see here dont need special treatment

Make all those changes then do more test flights

2 Likes

Thank you so much for the quick reply!

In mission planner on the tab for log download, there is a tab that I think its auto analyze, I believe that’s what it’s called, when I run that it comes back with magnetic interference and a GPS_glitch.
I was in a field with noting around. not sure why I had magnetic interference. (GPS is mounted on the back of the craft and is 4-5" above any electronics) Thats why I was thinking this cheap gps I have maybe an issue. With that said tho I’m not sure how reliable the auto analyze tab is. But I went ahead and ordered a holybro M9n gps.

For reference this is the FC I’m using:
Skystars H7 Single BMI270 Gyro FPV Drone Flight Controller - 30x30mm (pyrodrone.com)

And this GPS:
Amazon.com: Deegoo-FPV GPS Compass Module+ NEO-M8N+ GPS BDS Module Precision APM Drone Control, GPS Receiver FPV Flight Control Pixhawk Navigation Module for APM PIX PX4, Compatible with APM Port I2C MWC : Electronics

With the H743, I didn’t think that CPU load would be an issue, Tho I’m not sure how to check the cpu load either lol
Coming from betaflight, redundancy isn’t something I’ve ever really considered. Having more than one IMU really starting to make since atm… :face_with_peeking_eye:

But yes it was flying well until the instability, and it was when I let off the sticks (GPS would take over) that things went bad)

I was wondering about the FFT I could see that is was doing some good but didnt seem like it was doing all that it should be. and again, Not sure how to check to see how well its tracking.

I will check the 5V on the FC that powering the GPS. Ill note tho the only thing the FC is powering is the GPS the RX, and HDzero but that is coming from another bec (10V). the telemetry radio and LEDs are powered by an offboard bec.

I’ll make all your suggested changes and report back, Tho it may be later this week due to work.

Thanks again!!!

Once the copter is reasonably stable, do a flight with plenty of yaw, a circle and some ascent and descent.
Then we can improve the compass calibration using Magfit:
ArduPilot MAGFit

I checked the notch filter effectiveness and gave you those new values to use via the Filter Review tool:
ArduPilot Filter Review Tool

Hi Shawn,

So… I came across this today, And I am wondering if it could be associated with the issue I had.

BMI270 gyro tracking issue · Issue #11894 · betaflight/betaflight · GitHub

[ctzsnooze]
“I can confirm that the BMI270 has issues with gyro calibration, and that integration of the gyro data during rapid movements in level mode results in incorrect assessment of the movement of the quad, which is corrected once the quad slows down to less than 15 deg/s, causing a ‘jump’ in attitude.
At present Betaflight does not have a solution to write the calibration values back to these gyros.”

What are your thoughts, do you know of it being an issue here with ArduPilot?
Is there a way to write the calibration values back to these gyros with ArduPilot?

So… with that being unclear ATM … I have a different Fc headed my way. hopefully it’ll get here before this weekend.
Kind of going to be starting over with a new FC but I would still like to get your help with setting it up like we were going to do with the skystarts FC.

If you can help me should I start a new thread or can we keep going with this one?

Again, I deeply appreciate your help!

Edit:
I would call this a jump in attitude 100%

Hello, Shawn. Haven’t heard from you. I hope you’re still with me.

I was finally able to grab a log today. The weather was decent. The wind was acceptable.

This log comes from the same craft in exception to it now has a new FC and a new GPS. I applied all the settings That you had suggested.
I hope you can find the time to look over these and tell me if there’s any adjustments that can be made.
If you find this log to be insufficient. Please advise on what you need specifically and I will Try to record another log tomorrow.

2024-02-09 17-39-19.bin

Again, I appreciate all your help.

Good work.
Harmonic notch filter working well.
Attitude control seems fairly good, especially for not being comprehensively tuned.

You can set these now:

ATC_ANG_PIT_P,6
ATC_ANG_RLL_P,6
ATC_THR_MIX_MAN,0.5
BATT_FS_CRT_ACT,1
BATT_FS_LOW_ACT,3     // or 2 for  ordinary RTL
GPS_GNSS_MODE,7       // or 67 whichever gives the lowest/most reliable HDOP 
INS_ACCEL_FILTER,10
PILOT_THR_BHV,7       // spring-centred throttle
PSC_ACCZ_I,0.4
PSC_ACCZ_P,0.2

With the GNSS mode setting fight the urge to just look at the number of sats, lower HDOP is what you want. The reason I’m recommending to change this is because with the default setting the GNSS unit is trying to use all constellations and this is overwhelming and affecting the update rate.
A few years ago the GNSS units only did one or two constellations, so default settings were fine.

Are those 7" props?
If so you might want to try MOT_THST_EXPO at 0.60 and as low as 0.54

  • set too high you can see instability at low throttle
  • set too low you can see instability at high throttle

Once you are happy with the thrust expo, you could run autotune for some improvements.

Thank you for your reply.

I will set all these settings and do another flight in the morning.

The GNSS being overwhelmed makes prefect since, although I’m not sure how to see rather or not what mode gives the most reliable HDOP. so I will set 7 for now.

And no this is a 10" prop rig

I have never had good results with auto tune… If this was Betaflight I could tune it no problem, with just looking at the traces but… not having blackbox explore is killing me… To be honest tho… this flew pretty dang good just the way it was, for a GPS rig that is. race or free style would be a horse of a different color

MOT_THST_EXPO,0.60
should be good for 10" then, or up to 0.65

The HDOP is visible in MissionPlanner HUD if you add it by → right-click, User items, gpshdop
or in the status tab

ah okay thanks for the tip Ill set that right now.

also I have the radiomaster tx16s, so no spring-centred throttle

ok, acro people dont usually use the spring centred throttle, so ignore that setting of course.
In the log it looked like it was spring-centred. It’s also easy to change on the TX16

yeah, I do fly ACRO with all my other quads this is the ONLY one that I fly in that mode.

Thanks again for your help. I will report back tomorrow morning with a new log.

Here is the log from this morning’s flight…

2024-02-10 09-29-38.bin

It all went well. Although I took some notes during the flight.
One would be, it seemed like In attitude flight, The Attitude hold wasn’t quite as sharp as it was yesterday.
Also, in GPS hold, I noticed that it was yawing back and forth slightly. That it didn’t do yesterday.
And also in GPS hold. It was fluctuating up and down as well. There is a portion of that log there where it held GPS position for a while. While I was Observing the craft.

I also took note that it seemed like it had a little bit of oscillation in the motors, Especially in turns. That I don’t recall it doing yesterday. And it was windier yesterday than it was today.

Another note that the motors were stone cold when they came down today compared to yesterday being slightly warm. Slightly warm motors do not bother me. This is always an indication to where you’re filtering’s at. Cold motors mean You have sufficient filtering warm motors as you’re on that edge. Like UAV tech, Mark Spatz says. Filtering is just like porridge. Just enough is just right.

So I’m saying this leading to the question that. INS_ACCEL_FILTER,10 the documentation states: Filter cutoff frequency for accelerometers. This can be set to a lower value to try to cope with very high vibration levels in aircraft. A value of zero means no filtering

The question is, is this necessary? (INS_ACCEL_FILTER,10) Because We are doing more filtering. If I’m understanding this parameter correctly. And 2, we’re not dealing with high vibration levels.

So I’m curious if this has anything to do with the flight characteristics I am now observing. Or if my thought is completely off.

And as always, thank you again for your time and your help. It’s very appreciated.

I’d expect there to be some differences in flight mainly due to the Angle P increase and thrust expo change.

True, we can all agree on that :slight_smile: more filtering than necessary can introduce lag in the PIDs, filter real movements the FC should react to, or cause excessive CPU load for no good reason.

INS_ACCEL_FILTER,10 is the new default value we are using these days. It’s not changed to default in parameters, still at 20, because of the large number of big expensive copters using the old value of 20. Changing this to 10 during a firmware upgrade could have a negative effect on some copters.
For the rest of us 10 is fine, and a very good starting point. It’s not like the Gyro filter where you might increase it to get some better, more sensitive attitude control.
Usually with INS_ACCEL_FILTER,20 you would see slightly more nervous behaviour if your copter was sensitive to the accel filter value. Feel free to try again with 20 , as there’s nothing wrong with using 20.
The Accel filter would not usually affect motor temperature either, unless it was somehow causing excessive oscillations.

The filters that DO make the most difference to attitude control are the Gyro and harmonic notch filter, and yours is already working perfectly:

And the PID’s are what make the most difference to motor temperatures, assuming the motors are good for the thrust demands. In particular the Rate D term value, and even it’s proportion to Rate P.
I suspect the motors might have been warmer before because you had quite low (default) Angle P terms, and I think this was forcing the Rate PIDs to work quite a bit harder than they should, but we didnt have the PID logging turned on then.
Since increasing the Angle P’s I suspect the Rate PIDs have settled a bit.
In the most recent log the attitude control seems at least as good, if not better and smoother, which is very good given that there is more wind.

One other thing I compared (one of many) is the GPS speed when RC inputs are neutral, when position should not be changing. In the most recent log that GPS speed is fractionally higher or noisier during those periods, which could be from the wind, or even GNSS conditions (that vary from day to day)

earlier flight

Attitude control

GPS speed vs inputs - during neutral stick, average 0.06 meters per second

later flight

Attitude control

GPS speed vs RC inputs - during neutral stick, average 0.1 meters per second

GPS
The HDOP was slightly higher than what you previously had, try changing to using Galileo instead of GLONASS
GPS_GNSS_MODE,7

Also on the map of your flight, we can see there is some divergence of GPS position and IMU-calculated position. This happens mostly when you are near stationary, which will produce those near-oscillations you see in Loiter mode, as the copter tries to follow a wandering GPS position.

Compass
There is some improvement to be had in the compass calibration too. Magfit gave these values and it should be a very good calibration since you had a great amount of yaw and activity in your log.

COMPASS_DIA_X,1.02065
COMPASS_DIA_Y,1.00881
COMPASS_DIA_Z,0.97054
COMPASS_MOT_X,0.74812
COMPASS_MOT_Y,1.48353
COMPASS_MOT_Z,3.41530
COMPASS_MOTCT,2
COMPASS_ODI_X,0.00518
COMPASS_ODI_Y,-0.00619
COMPASS_ODI_Z,0.06606
COMPASS_OFS_X,-3.91972
COMPASS_OFS_Y,-23.32287
COMPASS_OFS_Z,10.54336
COMPASS_SCALE,1.01344

Thrust expo
I think the value you have now is correct. I could not see any poor attitude control during high or low throttle that might be caused by the expo value. In the previous log there was some deviation from pitch and roll that might have been because of the old expo value, but it is a very small difference.
A calm day may help to confirm or deny, but either way there’s no big problem and I think you have the correct value now.

In summary, I think attitude is quite good and running autotune would give very “locked-in” PIDs and attitude control on this copter.
Any unsteadiness you are seeing is likely because of the GPS position - if this continues we can relax the position controller slightly. That updated compass calibration may help too.

I am in your debt. I sincerely Appreciate you taking The time to write out a detailed explanation.

While I was waiting for your reply, I changed these back to there default.

PSC_ACCZ_I,0.4

PSC_ACCZ_P,0.2

INS_ACCEL_FILTER,10

I took another flight and the issues I was having before with the poor Altitude hold (Ascending and descending without any input from me.)during GPS went away. Also, the yawing it was doing during a GPS,and the sound i was hearing form the motors also went away.

Not sure which one of those did the fix, but. It wasn’t doing it before. Started doing it after I changed it, so I went back to those and it stopped. Perhaps you can offer some insight there?

I have just enough daylight and time to grab one more log. With the updated compass calibration numbers you gave me. And changing the GPS GNSS mode to 7

I’m going to change INS_ACCEL_FILTER,20 Back to the 10 on this log. To see if I have any of those flight characteristics come through like I did before. Maybe that will help narrow down what caused The yawing and Ascending and descending.

(I went ahead and ran out to gab a log before posting a reply)

OK, so I went out. And this log here 2024-02-10 17-39-21.bin
is with INS_ACCEL_FILTER,10
And I immediately noticed the oscillations back. I don’t know if oscillations is really the right word to use here, but I could just I can hear it in the motors. Speeding up and slowing down.

Immediately went into the house and changed it back to INS_ACCEL_FILTER,20
Took off again and those sounds went away, so it has something to do with that.
This log here 2024-02-10 17-46-24.bin is the log from that flight.

And a side note: I did have the pid logging turned on … I think?

Again and as always, thanks for all your time and help. If you see any improvements with these logs. I can implement that before I attempt to try auto tuning again. I’ve tried it before and had poor results. The P value on pitch was set way too Low and had some very abnormal flight characteristics That nearly ended up into a crash when I was testing it after auto tune was done. Any advice in that area would be Certainly welcomed.

Good work for methodically trying those changes and providing logs, thanks.

I had an idea that PID logging wasnt enabled in the earlier log but I could be wrong, and probably missed something. I’ll look in the logs again because I’m interested to see what the difference was when you increased the Angle P.
I was swapping between log files a lot and trying to compare many more aspects than what I mentioned.

Back some time ago we only ever decreased INS_ACCEL_FILTER below default if there was a vibration problem. So there is nothing wrong with using a higher value and it seems to suit your copter much better. I’m going to keep checking in your logs to see where I can identify what you are observing, and how things change with different values of INS_ACCEL_FILTER for my own information and just in case I can provide further insight.

I want to look into the altitude issues too, since the accel filter would have little to nothing to do with that. Quite often with any amount of wind there are pressure changes that affect the barometer quite a bit, so I’ll look for that too.
Those PSC_ACCZ parameters do affect altitude, and smaller copters could use more aggressive values than bigger copters - same for ATC_THR_MIX_MAN too.

It’s likely that Autotune produced a bad result due to some interference from noise, more often than not autotune does a better job than manually tuning. It wouldnt hurt to try again because it quite often arrives at PIDs that you would not have tried yourself, and you can always undo the changes if you dont like the result.

I’ll let you know what I find - until then get out and fly :slight_smile:

I looked at the logs a little out of order, but anyway…

Log 2024-02-10 17-39-21

INS_ACCEL_FILTER,10
PSC_ACCZ_I,1.08
PSC_ACCZ_P,0.54

There is an altitude controller oscillation in rate (output) and P & I , not making it into the actual altitude except that there seems to be very relaxed altitude control.
However this oscillation is making it into the motor outputs.

Log 2024-02-10 17-46-24

INS_ACCEL_FILTER,20
PSC_ACCZ_I,1.08
PSC_ACCZ_P,0.54

Basically the same as the previous log, altitude controller oscillations are making it into the motor outputs, so it’s some sort of luck (or below some threshold) that you are not hearing the oscillation.
Altitude control is slightly better but still oscillating around the desired altitude

Log 2024-02-10 09-29-38

INS_ACCEL_FILTER,10
PSC_ACCZ_I,0.4
PSC_ACCZ_P,0.2

Larger slow altitude oscillations are obvious. The high frequency oscillations in the altitude controller are reduced in amplitude. The altitude controller oscillations are not noticeably making it through to the motor outputs.
Any audible motor noise should just be the general changes in motor outputs as the copter hunts above and below the desired altitude. Of these three logs this is the least similar to the motor noise we get from oscillating outputs, like when D term is too high.

The Cause
This something I missed until now, and something many others probably would have picked up right away.
In all three logs the barometer is a significant contributor and altitude changes are exaggerated. I would say prop wash over the barometer is the major problem. The barometer noise and fluctuations are adversely affecting the altitude controller.

I admit I’m used to looking at flight controllers that are fully enclosed and dont usually exhibit such bad barometer problems caused by prop wash, at least not as much. While going over what I missed, PID logging was enabled in the earlier log and I missed that too, I was probably trying to look over too many items.

The Fix
Cover the barometer in open cell foam, maybe see of there is a cover that can go over the body of the copter to keep light and prop wash away from the flight controller.

I would keep these values and try to work with them. We have a large number of copters out there working extremely well with these values (given that the ACCZ values can be linked to hover thrust)

INS_ACCEL_FILTER,10
PSC_ACCZ_I,0.4
PSC_ACCZ_P,0.2

We could also add some PSC_ACCZ_D and reduce PSC_ACCZ_FLTE if required, for example if the barometer output cant be solved by physical means. There is also EK3_ALT_M_NSE if we need it.

And unrelated, but also increase yaw control a bit:

ATC_RAT_YAW_I,0.05
ATC_RAT_YAW_P,0.5

This barometer issue would also explain a bad result from Autotune, and I maintain that Autotune should do well given the otherwise excellent performance of this copter, and especially once the altitude issue is resolved.

Wes, let us know how you go with this, I’m extremely interested in the outcome.
Next time we will call in Dave @dkemxr and save ourselves a few hours, but I’m also welcoming any learning I get from this.

Well, it would seem that we both overlooked something, so I guess that makes us even. HAHA

I completely forgot about covering the Biro up… I knew better. I know better. And I even covered it up with some foam on the last flight controller. And I even told myself a couple of times I need to remember to do that, but in the haste of everything and being excited to get this thing in the air finally. It completely slipped my mind. This flight controller is laid out a little bit differently than my last one, so. Where I was able to put the foam to cover the biro it isn’t going to work with this new FC, but I devised a plan…

I designed and made a 3D Printed FC cover. and for good measure i even cut a piece of open cell foam and laid it over the top of the Biro as I put the cover on. I am confident this will Fix the problem with air and or light affecting the Biro.

After I read your e-mail, I was thinking about the noises I was hearing from the motors as it was flying. And after giving it some thought about when I realized I heard those Sounds from the motors was when the copter was leaning into the wind. With a south wind and I turned into the South wind, which would expose the biro even more to turbulent air Than if I was heading north with the wind.

So I feel confident that your diagnosis is the culprit.

I will set these back and do another flight once I get this all buttoned back up.

INS_ACCEL_FILTER,10

PSC_ACCZ_I,0.4

PSC_ACCZ_P,0.2

Thanks for the suggestion about Yaw as I Had noticed that it needed some work. Rates turned up or something. it felt slow and sluggish would be my best description. So I will apply those as well.

Unfortunately it is going to be raining and eventually turning to snow all day today. So it may be towards the weekend before I can get another flight in.

But I will promptly post the log just as soon as I get one.

As always, I appreciate all your time and help. It’s nice to know, even in the world that we live in today. That we can still rely on fellow hobbyists for help.



Nice little cover!
I’d seen the yaw performance all along, but left it alone until I thought we had a fix for the problems.
Yaw can often be left until last.

Good day, Shawn., I slipped out of work early today and came home and got a flight in. It was decently windy, windier than what I would normally fly in, but I wanted to get that log.

Huge improvement. I don’t hear the stuttering or shimmering or Oscillation in the motors at all! In fact, it sounded smoother than I have ever heard. No problems with Attitude control. And the Yaw control is much better. I do have a question though. Is there any way that I can increase Ascend RATE? I know there’s a parameter for it and I’ve messed with it some, but… It still seems kind of wimpy. And there’s a lot of –Slop- in the middle of the stick before it starts going down or before it starts going up. Is there a way we can fix that, Also? I’m an Acro pilot by nature. And having that much throw on the stick before it goes up or down is kind of…Annoying to be honest. Lol.

So after you look at this log 2024-02-14 17-47-29.bin and if there’s any other suggestions you can see, I will apply those before I attempt an auto tune again. I don’t know when I will get to that exactly, it really depends on when I have decent weather.

Thanks again for all the help!