Quadcopter fly away

[Notorious7 (just_curious)]

I am using arducopter version copter-4.0.2 on a quadcopter. I am using matek f405 ctr flight controller. I am flying and controlling quadcopter using a self made GUI(GCS) which sends mavlink commands to a companion computer mounted and attached to quadcopter. I am using RTK HERE+ GPS.

I have fly this quadcopter many times before and it was flying very well without any issues. But today i have to terminate the flight mid-air cause quadcopter went out of control and it kept increasing its height maintaining the same position.

When i checked the logs, i found there was gps glitch,followed with ekfcheck-2 error and some error mentioned as err:29-1 which i am not familiar about and never seen anywhere before in any ardupilot documentation.

When i review the log, I checked BARO.alt,GPS.alt and POS.alt, i am not able to understand why there was a gps glitch in the first place. i also checked NKF4.sv and that was also out of the acceptable range during my quadcopter went out of control. I think there was some issue from both gps and barometer end.

Another thing is that there is no log of POW.vcc, i ensured flight controller getting right power from the input end but i am not able to ensure that during flight from logs, i doubt if gps getting sufficient power or not, and that might be causing gps glitch.

I am attaching my quadcopter fly away flight log here, where there is record of 2 flights in the same log, i have seen slight misbehavior in the first flight too but quadcopter fly away in the second flight and quadcopter went out of control, after that i terminated the flight using mavlink command and quadcopter crashed badly.

QuadCopter Fly Away log

Can anyone please help me figuring out why these things happened and what was the main cause of fly away, whether the reason for fly away was avoidable or not.

[xfacta (Shawn)]

My money is on vibrations, check the clipping:

The usual fixes apply, prop balance, wires transferring vibes to the FC…

Set up the battery monitoring, both voltage and current, and then do a Compass/Motor calibration too.Then set these:
MOT_BAT_VOLT_MAX
MOT_BAT_VOLT_MIN
They allow Arducopter to scale the PIDs based on battery voltage and better control the motors - not exactly related to your issue but good to do anyway.

[dkemxr (Dave)]

What’s interesting about this log is the Vibe levels are not that high yet there is clipping accumulating. You also had a Vibe level FS action taken (that Subsystem ecode you mentioned).This changes the altitude controller and may be responsible for the crash. I have never had one of those, the feature is relatively new.

[rmackay9]

Thanks very much for the report. As @dkemxr says the vibration failsafe is quite new and this is actually the first time that I’ve seen it trigger on a real vehicle.

EDIT: originally I thought the vibration failsafe triggered before the vehicle started climbing but upon further
investigation we can see the motors rose to a high level before the vibration failsafe triggered. What’s also clear though sadly is that the landing detector triggered, this is what caused the vehicle to crash to the ground.

While we investigate it might be best to disable the vibration failsafe by setting FS_VIBE_ENABLE = 0.

The EKF is definitely unhappy (which is why the vibration failsafe kicked in). The graph below shows the three values from the EKF that are used to trigger the vibration failsafe. These are:

• Vertical velocity “innovation” (NKF3.IVD) must be positive
• Position “innovation” (NKF3.IPD) must be positive
• Velocity “variance” (NKF4.SV) must be under 1

It’s operating as designed so I think I’ll need to discuss a bit with our EKF expert, @priseborough

By the way, I wouldn’t classify this as a “fly away” because the vehicle stayed under 50m in altitude and was never more than perhaps 10m from home - it actually never lost horizontal control. I think in general the “fly away” term is reserved for vehicles that lose control and fly off a significant distance and/or can’t be found.

[priseborough]

The problem with clipping is that it drives the average Z accel reading to zero which makes the inertial nav inside the EKF think it is falling. This can result in a series of repeated uncontrolled climbs as the height controller attempts to correct what it sees as a large loss of height and the EKF periodically resets the height and vertical velocity to the Baro/GPS.

This was an appropriate activation of the vibration failsafe. The issue to follow up is why the landing detection triggered. Given the likelihood of unreliable vertical state estimates when a vibration failsafe is active, we may need to modify the landing detection for that scenario.

[rmackay9]

After looking at the logs a bit more we’ve found that the user sent a Flight Termination command to end the flight. The appearing of the landing detector in the logs comes after this and is expected.

Below is the MAVC command with CmdId 185 which is Flight Termination.

So we can investigate a bit more about whether the vibration failsafe did a good job or not but it looks like the vehicle lost altitude control because of high vibrations.

[Deficiant_Codger (De_Coder)]

Yes we sent flight termination command because if it has gone away from our network reach we can’t able to do anything other than losing our quad and yesterday something very horrible happened we did disable vibration failsafe off we ensured that even at very high throttle sensor don’t have much affect of vibration by making quad frame as rigid as we can but when we takenoff by mav command and setting alt to 2 meter it takes off and flyaway to abnormal height it goes to far away from network to make a land command and also i would like to add that the quad even refuses mavlink’s land command, so we designed a kill switch which restart the entire flight controller and when we press this all motors stop and drone crash to the ground. I would like to say that in your previous comment you said don’t use word flyaway coz it didn’t flyaway actually that’s not the case we just hitted kill switch early as soon as it raises to abnormal height that’s why it’s not gone, but as commanded by you we disabled vib failsafe and taken off again and merely after 8-10 second it takes too much height and we didn’t hit kill switch and in 15second it goes off from network and gone too much height to be seen after few mins it comes crashing too the ground any patch to it we already lost our quad coz it’s badly crashed to the ground after battery drained completely damaging everything which was way to expensive, don’t want to loose another one

[dkemxr (Dave)]

Your only control is via a GCS? Did you recover a log file from that crash? You should have addressed the vibration problem before flying again.

[xfacta (Shawn)]

I think everyone would highly recommend using an RC transmitter and receiver at least until all tuning and extensive testing is completed. Get up a bunch of flight hours and reliability, get Failsafes configured and tested, then decide if the GCS-only configuration is worth the risk.
Maybe change to using the RFD “X” series telemetry radios - you get range, reliability and RC control all in one. The RC control over the RFD radios would have more latency than regular RC.
There’s even smaller units if space is an issue.

Edit, fixed error "RC instead of “RFD”

[Deficiant_Codger (De_Coder)]

everything is damaged badly sd card i don’t know how but corrupted no logs can be recovered. And as i said in previous replies we did addressed vib. issues but i tried best to stop all vibrations and yes i am sure there is not much vibrations. Also like to add before testing any drone via GCS we did fly it with radio tx rx on POS. HOLD mode and they fly pretty well.

[Deficiant_Codger (De_Coder)]

yeah we are thinking the same but still wants to know why this flyaway being triggered i don’t know why these are causing

[xfacta (Shawn)]

OK tell us a bit more detail about the quadcopter and components used, maybe a photo or two.
See if you can hover for a while and provide a new .bin log
And just to reiterate:
Set up the battery monitoring, both voltage and current, and then do a Compass/Motor calibration too.

[dkemxr (Dave)]

Then all you may have is a Tlog from the GCS. Not as much value but maybe better than nothing.

I don’t see any replies after the log you posted showing vibe clipping that you corrected this.

[Deficiant_Codger (De_Coder)]

Component being used
1.Matek f405 Std with arducopter 4.0.3
2.Here+ Rtk Gps
3.1700kv motors
4. 4 in 1 esc
Matek f405 std has no onboard cuurent/volt sensor so can’t set battery monitor
i will try to provide a clean log asap

[Deficiant_Codger (De_Coder)]

Sdcard is showing an APM folder but no matter what i do it says i/o error

[dkemxr (Dave)]

The Tlog is on your GCS device.

[Deficiant_Codger (De_Coder)]

our GCS is not mission planner it’s a custom built gcs using pymavlink don’t know where logs are or what even they save

[Deficiant_Codger (De_Coder)]

we just disabled vibe failsafe but now don’t have logs to check either again a failsafe of vibe triggered but how can it be triggered if it’s been turned off

[dkemxr (Dave)]

Disabling the failsafe isn’t addressing vibes, you have a mechanical vibration problem causing accelerometer clipping as shown on the graph above. Prop balance, poor vibration isoaltion of the Flight Controller, compliant frame all could contribute. Here is another view:

[rmackay9]

@Deficiant_Codger,

OK, sorry for the crash. I probably should have updated my advice after we discovered that the vibration failsafe was not the cause of the problem. The basic problem is the vehicle has high vibration and clipping and this causes the EKF to be able to calculate a good climb rate. The vibration failsafe probably would have helped keep the vehicle under control - it probably just needed a bit more altitude to retake control.

I agree with the advice from @dkemxr and @xfacta - the core issue is high vibration in the frame and it would be safer to fly with an RC until these problem are worked out.

The dev team doesn’t normally make specific recommendations on flight controllers but it might be a good idea to switch to a flight controller with built in vibration isolation. Alternatively be very careful to add proper vibration isolation to the frame (wiki page here on the subject).

Re the custom ground station - I think it might be good to add the EKF status and vibration status to the GCS to help the pilot be more aware of problems. The Mission Planner has a convenient display of the EKF health and Vibration levels.