7inch Quadcopter AC 4.1.0-Beta5 : inflight EXK3 lane switch : Root cause Analysis request

Hello,

I have build a 7 inch LongRange quadcopter using :

  • Armattan Chameleon TI LR carbon frame
  • Matek H743 SLIM running Arducopter 4.1.0-beta5 with bidir dshot firmware
  • ESC : T-Motor F55A Pro II T-Motor - 3/6S -BLHeli_32 4in1 MCU F3
  • Xing2 2506 - 1650KV motor
  • Propeler : HQ Durable Prop 7 X 3.5 X 3V1S
  • External GPS & COMPASS : M9N-5883
  • External Lidar + Flow Sensor : Matek 3901-L0X
  • RC : TBS CrossFire Nano RX with Yaapu script telemetry running on TX16S
  • LIPO Battery Tattu 1800mha 100C

The quadcopter suffer from inflight EXK3 lane switching, which generate drop in height and fast attitude correction… which are quite scary from the pilot point of view…

May be I should tune EK3_ERR_THRESH by adding +50% … but i don’t want loose the vehicule…

I need help to analyse BIN LOG and find the root causes !
BIN LOG : https://drive.google.com/file/d/1olcCFbF7xcIIrjBJCvZokKpq5jDBpmmN/view?usp=sharing

I can see in the logs peak in XKF4.SV before/during the lane switching, but there is lots of XKF data to be analysed…

In general, does adding more sensors ( 2nd GPS, 2nd Baro, 2nd IMU…) would help EKF3 to better manage position estimation ? raw sensors value noise ?

Would adding GPS Blending (aka Dual GPS) help ?

Would adding an AP_Periph like Matek - GNSS M9N-F4-310 help ?

My intend is to have long range flight with very stable position holding .

Thanks in advance for any support.

How about disabling batch logging so these Logs are not so huge? The notch filter is already set. INS_LOG_BAT_MASK to 0.

Let me know how you like the build - seems like a sweet combo

Weel, it seems that dave is right : I will disable those extra logging and post a new flight bin log.

Anyway any advise to get a more stable EKF3 positionning ?

I’ll take a look but I’ll add I see this occasionally with a Mini version of that FC on a 5" and don’t sweat it.

And as Dr. Piper @andyp1per says that is a cool bit of kit you have!

Hello,

New 10 minutes flight with only one EKF3 lane switch 0->1 . sorry could not get more this time.
BIN LOG : https://drive.google.com/file/d/1OrA9_7ZFJm5Pz0GFlTytj9tfp9uUGdjk/view?usp=sharing
Size : 102 Mbytes

For fewer BIN LOG files, i will switch from ATTITUDE_MAST to ATTITUDE_MED on next flight. Only had 1 EKF3 lane switching…

Thanks very much for testing and providing feedback.

The jerk in the vehicle comes from the altitude reset not being handled properly which is a known issue and exists in Copter-4.0.7 as well. We should resolve it of course and it is actually not that difficult to fix … still I can’t promise when we will fix it.

The attitude change is very small and is handled properly though which is nice to see. See how there’s no jerk in the rate output (the lowest level rate controller output) for the three axis? This looks good I think.

Thanks again for the report and highlighting that it is important we get the EKF reset handled completely.

I have this 15inch small copter with FW 4.1.0Beta5 and a old Pixracer in it. After I put in a new SDcard I have a short log file now for you if you find some time.
A very short flight test in my garden. At the end a small crash. It hit a leg of a garden chair. No big deal - one prop only.
But this Quadcopter does not like 4.1.0 yet,
PS ( I am testing with one of my eyes only)Just can’t wait until after my eye-operation in 4 weeks. That’s why a stupid crash on the ground. No other flying at all at the moment! https://www.dropbox.com/s/fixd2dtr6hm6mdl/2021-07-11%2013-45-44.bin?dl=0

@FRED_GOEDDERT,

Txs for the report and log. The vehicle’s control looks pretty good (ATT and RATE messages) although the yaw is oscillating a bit but the numbers are quite small so I guess it is a small fast oscillation.

Anything in particular you’re noticing?

Thanks for looking at the bin-file.
I noticed that the copter is very unstable in the Position-vertical!
I read in the Missionplanner replay of the .tlog file Error pos vert variance.
It hovered perfectly in stabilize but hard to control in Z at Alt Hold.

Ah, yes, there is quite a bit of oscillation on the Z-axis in AltHold mode.

We’ve got some advice on tuning the vertical position controller here on the Tuning Process Instructions page.

Looks at the log it looks like hover throttle is 0.33 so params values may help:

PSC_ACCZ_I, 0.66 (was 1)
PSC_ACCZ_P, 0.33 (was 0.5)

Maybe @Leonardthall or @andyp1per has more advice…

Hi Fred,
None of these things I list is critical, but interested to see how they affect the next flight. The most important settings will be those Randy mentioned.
INS_GYRO_FILTER , 50
ATC_RAT_PIT_FLTD , 25
ATC_RAT_PIT_FLTT , 25
ATC_RAT_RLL_FLTD , 25
ATC_RAT_RLL_FLTT , 25
ATC_RAT_YAW_FLTT , 25
also these harmonic notch filter settings can be changed a bit
INS_HNTCH_REF , 0.33
INS_HNTCH_ATT , 40
INS_HNTCH_BW , 51
INS_HNTCH_FREQ , 102


After a few rainy days I did another short test hover with may copter. I changed the settings Randy mentioned above and also I put all your values in.
Test flight - Absolutely no difference in the behaviour of the copter - not good at all.
I than went back to the Firmware 4.0.7 stable. 2 hours later I did a new test flight, same spot in the garden, calm weather and the drone behaved normally, good.

What I found out was, the compass failed in FW 4.1.0 Beta5 - mag_field interference was 80.74% and the other flight 16.0% off.
But with FW 4.0.7 stable the Compass did not fail - interference within limits -3.69%.
That is strange because that was my last test without doing nothing to the copter -just loaded the stable version 4.0.7 back over the Beta 5 - 4.1.0. Same spot - same condition.
My copter does not like the 4.1.0 Beta 5 version.

@FRED_GOEDDERT,

I guess you’re using MP’s auto analysis and it is just complaining that the compass offsets are too large. This is not really a problem and we should probably remove that check from the auto analysis.

A more useful check is MP’s “Sensors/Compass/Compass vs Throttle mavgraphsMP” which shows that there is very little interference on the compass from the motors which is a good thing.

So the primary issue you’re seeing is that the altitude is unstable? Can you be more specific? for example is it slow drifting in AltHold mode or maybe you’re hearing the motors jump up and down? Is it occasional severe jumps? Sorry to be annoying I just need some more information to narrow my search through the logs.

The altitude hold performance in AltHold mode looks ok in general because we see the desired and actual altitude tracking well.

I suspect what you’re seeing is the occasional jumps in the altitude because of the EKF core change. Here are the motor outputs when this happens so you’d notice this.

It might be a good test to try disabling the second EKF core by setting EK3_IMU_MASK = 1 that will get rid of the switching although it will also mean you’re flying with no backup IMU. Still, IMUs very rarely fail and this may help us narrow down on what the issue is.

Any additional clarity you can provide on what’s not good would really help.

You are right I got that compass info from MP’s auto analysis.
The copter is drifting in Alt hold and it is not holding altitude at all. Going up and down on its own in a slow but nerving manner.
But it could be - suddenly it is going faster up or down and I have to quickly compensate harder with the stick. It is not a real jump, so the motors are not screaming. Because I have very little space in my garden ( house and big trees around) I have to interfere immediately.
If you look at the flight with FW 4.0.7 I did after the one you viewed you see how stable the copter is in Alt Hold and Loiter. (Slide drift need maybe Accelcalib. - levelling.)
Hopefully that helps a little.
I will load FW 4.1.0 beta5 back into the copter and test with setting EK3_IMU_MASK to 1 and report back.
We have winter time and weather is not suitable again at the moment. Will come back later.

@FRED_GOEDDERT,

Here’s a graph of the desired vs actual altitude (scale on left) and the output throttle (scale on right) for 4.0.7 vs 4.1.0-beta5.

What is most obvious is the barometer altitude perhaps has more sharp changes in 4.1.0-beta5. The flights are done at quite different altitudes though as well (30m vs about 2m) so it could be environmental.

In general I’d say that 4.1.0.-beta5 is doing a better job of maintaining the target altitude but perhaps because it is more active.

It’s really not clear to me if this is an environment issue (e.g. larger air pressure change at high altitude) a sensor problem (e.g. interference from the motors on the barometer), an estimation problem (e.g. something to do with the change from EKF2 to EKF3) or a control issue (we introduced a velocity I-term.

I’m afraid we may need to ask you to do a few tests to help narrow down the problem:

  1. fly back-to-back tests at the same altitude with 4.0.7 vs 4.1.0-betaX
  2. enable EKF2 on 4.1.0-beta5 by setting AHRS_EKF_TYPE = 2, EK2_ENABLE = 1, EK3_ENABLE = 0 and reboot the autopilot

@FRED_GOEDDERT,

@Leonardthall and I had another look at your logs and it looks like the problem is the “sensor” issue. In particular have a look at the bottom picture (Copter-4.1.0-beta5 above) and check out the huge changes in barometer altitude at the end of the log. The barometer altitude is change by more than 25m!

This barometer issue should not be related to the version. It is most likely environmental or possibly something to do with the frame design. I suspect it is the latter and the barometer is very exposed.

To confirm it is the barometer you could try setting EK3_SRC1_ALT = 3 (GPS) to see if that helps although the real solution is probably to protect the barometer more.

The other problem you may be facing is your vertical velocity filter settings:
PSC_VELZ_FLTD 20 Hz
PSC_VELZ_FLTE 20 Hz

So this may be part of your issue too.

Thanks for your suggestion. I will change that Leonard.

Randy
Regarding Barometer. Well, I had that old Pixracer 5 years in a plane, very little use.
I definitely will do a extra cover up now being installed in that copter recently.
But here is a clue regarding the height of my flying. In both test with both firmware I was flying
between 1 and 3 meters max. FW 4.0.7 was pretty right in height.
The tests with FW 4.1.0 beta 5 was showing always much to high ( yesterday and 9 days ago as well with beta5)
But I will see if the Baro is the problem.

1 Like

I have covered the baro at the Pixracer and finally (after my Eye-Op 2 weeks ago)
I did today a short test hover in the same area but a little wind from the left.
Using the latest Beta 7 , I found no problems even down in the prop-wash.
Max height was about 3 meters.
Thanks for sorting out my problem. - https://www.dropbox.com/transfer/AAAAAGGJbB22RfF2vg34u781VcQvIlXj3ZVbjECFoD9fEZAhsK6ieqE

1 Like