Autotune Mishap with AC3.5 rc 2

Hi All,

I decided to load up 3.5 rc2 onto my quadcopter. This quad flew pretty decent on 3.4 but I wanted to try the new autotune in 3.5 to see if it dialed things in a bit more. I tried to fly it to run an autotune last night but when I flipped the switch to start autotune, nothing seemed to happen. Since it was dark, I chose to wait until daylight to try it again. Today, I tried autotune again. At first it seemed as if nothing was happening. I turned it on and off several times trying to get some sort of action. After leaving it in autotune for a while, it did seem that it started to react. i thought it might have been a new intentional behavior. It quickly got very twitchy and then violently flipped itself upsidedown and propelled it into the ground. Not a fun moment. Still a complete novice in log analysis, so dont know what to look for. Can anyone tell me what may have caused this behavior, especially since 3.4 flights and autotune never yielded anything like this?

00000360.BIN (1.7 MB)

did you happen to have a GCS running? I’d love to see the tlog of the flight if you did

we notice channel 9 goes high at the end, which is emergency stop. Did you trigger emergency stop after it flipped over or before?

Yes I was. Here it is. 2017-03-20 18-19-07.tlog (440.5 KB)

As soon as it flipped over I flipped my emergency stop switch to try to save motors and props. My guess is I hit the switch about when it hit the ground.

Anything helpful in that tlog?

Jimmy,

Tridge, Peter and I had a look at the log and the issue doesn’t appear to be with AutoTune. The issue is that the timing of the main loop starts getting worse from about 29000 and the main loop actually stops for about 0.6 seconds at the time of the crash (around line 39000).

This is very odd and we don’t understand the cause yet. I’m not sure we have enough information in the logs to narrow down where the problem has come from.

For me, this is actually the most concerning issue we have with AC3.5. It has caused a crash and it’s unexplained…

Thanks @rmackay9,

The curious thing is that I flew about 15 minutes the night before with 3.5 rc2 with no problems other than autotune failure to commence. I engaged autotune several times but when it didn’t do anything I turned it back off. The time it crashed happened only after autotune was left engaged for a while to see if it would actually start the autotune process.

I am obviously reluctant to fly again as I would like to avoid additional damage. Could I do something different with logging to give you the info you need?

This is the first time I’ve started looking into the Loop time logs and maybe I’m interpreting the data incorrectly, but I’m seeing greatly increased NLon on 3.5-rc2. On RC1 my Mean NLon value was 71, with a max of 95.

On my last 2 flights with RC2, including the one where I filmed the successful AutoTune, my NLon Mean is 418, with Max of 468. I don’t have any spikes and haven’t had a crash, but definitely concerning to see the increase.

I’m running a Pixhawk 1 (clone, but seems to be working alright) and I believe I have all logging turned on, so that may account for the high NLon values.

PS, I manually installed the v2 version of the 3.5-rc2 firmware.

1 Like

John,

NLon (“Number of long running loops”) of >400 is quite high. That means basically 10% are running long. Running “long” means the loop takes more than 3ms to complete. Loops should normally complete in about 2.5ms so that we can achieve 400hz.
Do you have any extra sensors attached to the flight controller? lidar, optical flow, etc?
It would be interesting to see the effect of turning down logging and also to know what your LOG_BITMASK is.
I’ll have a peek at your log files if you like.

Thanks! Without seeing this thread I would have never looked into this since my flights on both RC1, and RC2 after a successful auto-tune have been some of the most stable I’ve ever had.

I’m attaching a link to a folder with 3 log files to hopefully compare the performance between RC1 and RC2 with regards to loop times. Current Log_Bitmask is set at 174078 so each log is too big to upload directly.

LOG FILES

As far as the full setup and sensors, here’s the rundown:

  • S550 Hexacopter Frame
  • Pixhawk Clone w/ M8N GPS/Compass
  • External LED/Buzzer/Safety Switch
  • SunnySky x2212 980kv Motors (9x4.5 props)
  • BLHeli 20A Opto ESCs
  • Mauch Power Module/Sensor
  • FrSky X8R Receiver w/ Taranis
  • Craft & Theory FrSky Telemetry
  • 3DR Telemetry Radio
  • 1.3" I2C Display
  • Zhiyun Tiny-2 Gimbal + GoPro Hero5 Session (Controls from X8R, not through Pixhawk, yet)
  • 5000mAh Battery

@jimmydaux2 I noticed you have AUTOTUNE_AGGR still at 0.1 but I haven’t seen any specs of your copter, although I noticed the RCOU was only in the 1400 range.
So I am assuming that it is fairly over powered.

I had an Octo flip itself into the ground on autotune because I left AUTOTUNE_AGGR at 0.1
It is running T-Motor U5 with 15" props.
Tuned now at 0.75 and it is very aggressive in manual.
I will be tuning them (there are 2 exactly the same) with AUTOTUNE_AGGR at 0.5 (recommended minimum) to tone down the aggressiveness as now a <1sec full stick deflection will just about turn the copters over.

Quadcopter is running 580kv motors with 15 inch props on 4s power. It is not overpowered by any means, but it has plenty of power for its intended purpose. I have run autotune countless times with this hardware configuration on previous firmware with no issues whatsoever.

Also, when I engaged autotune, it didnt actually start the dance. It did nothing for a while then started twitching, then did a really aggressive roll and then finally a power flip into the ground. It didnt resemble anything that I have ever seen autotune do in the past.

@rmackay9, any advice on this? Do you recommend switching back to 3.4 until you guys are able to identify the problem?

Jeremy

I’m another who tonight Autotune on AC3.5-rc2 would not start. I must have not let it go as long as Jeremy. Maybe 30sec to a min. No crash. My logs may shed some light. My LOG_BITMASK is 176126 and I’m running a LIDAR Lite V3.

First flight I engaged autotune while in poshold. Toggled it at least a couple of times. I had the axes set to all (7). No response from autotune. Log:
00000026.BIN (2.5 MB)

Second flight after axes changed from all (7) to Yaw only (4). Engaged autotune while in poshold a couple of times. I then tried althold, Still no response from autotune. Log:
00000027.BIN (2.6 MB)

My hex is on the larger end being 810mm with 15" props so I thought to split the difference for AUTOTUNE_AGGR @ 0.075. Here are my parameters:
3-21-17 Autotune Enabled.param (15.7 KB)

Bob

Jeremy,
We havent figured out how we are going to get to the bottom of this yet but we will discuss in in our tue morning meeting.
Your board could be quite valuable to our testing if the problem happens regularly. I was wondering if you could do us a favour and clear out the log files and then set the log-disarmed parameter to 1, then leave the board running for a long time (30min) and then post the log?
I could also make a special firmware that displays a message on the HUD if the performance gets slow. We could use that to see if the problem is repeatable on your board.in fact i suspect we will add this as a new inflight warning.
Thanks for your help on this.

Bob,
Your vehicle’s issue is actually very different from Jeremey’s.
I havent looked at your the logs you’ve posted yet but normally the reason it wont start autotune is because the pilot’s sticks are not perfectly centered so it thinks the pilot wants to reposition the vehicle. Could you check the transmitter’s trim tabs and try an RC calibration?

Ok, I will check. Thanks

@rmackay9 I’d love to help with this. Would you like me to do the custom firmware for this experiment or with my current build?

@johnnyruz,

@Erik_Groeneveld figured out the cause of your performance issue - both EKF2 and EKF3 are enabled. I need to check with Tridge and Paul but we may not be load balancing properly across the EKFs. The EKF3 is off by default for AC3.5 so hopefully this issue won’t affect the majority of people. It might be interesting to set the EK2_IMU_MASK to 1, EK3_IMU_MASK to 2 (or vice-versa) to see if that helps. That will ensure the 2nd EKF uses the slightly lower speed IMU.

To be clear, this is not related to the far more serious issue that @jimmydaux2 has reported at the top of this thread.

Thanks @rmackay!

I’m going to load 3.5rc3 tomorrow and do several short test flights while playing with the EKF parameters and take a look at the NLon results.