Im using Cube Orange FC with custom made Airframe. The copter has been flying really well with almost 140 flights and 42 hours of successful missions. Recently we encountered strange behaviour with Multiple GPS glitches errors and copter crashing in the mountain. Please look into the log for analysis and help me find the root cause.
Thanks
Log File: 00000141.BIN - Google Drive
Really high Z vibrations. Vibration fail safe was triggered. Motor outputs were hitting max. Is it possible a prop came loose, or something mechanically failed in an arm?
It’s worth upgrading to the latest stable firmware. The messages around GPS glitch have changed. Sometimes a GPS Glitch is really a compass issue or other issue.
But there are no clipping in Vibe msgs. The problem is not in vibration as we have many successful flights with the system. Pl check POS data it is showing too much abnormality!
@rmackay9 @hosein_gh @dkemxr
Your expert opinion needed please!
It looks like this entire Flight was under Vibration Compensation from the initial take-off (which takes nearly max motor output). EKF failsafe triggered the Vibration Compensation it doesn’t require clipping events. I’m not sure why the motor outputs were maxed out producing a thrust loss error but you don’t have much of a thrust/weight margin. The MOT_THST_HOVER value of .459 tells you that.
Update to latest Stable 4.0.5 is very old.
As I said, very high Z vibrations and if you check the messages you will see vibration fail safe.
2023-04-08 02:51:18.013 EKF primary changed:1
2023-04-08 02:51:19.079 Error: Subsys FAILSAFE_VIBE ECode 1
2023-04-08 02:51:19.079 Vibration compensation ON
It is my understanding that vibration levels under 30 are considered good and under 60 are considered satisfactory. The vibrations levels are not that bad for this abnormal behaviour. Don’t you see glitches in GPS data both Altitude and Lat/Long. Is it possible having glitches in GPS led to this event? The copter was in Loiter Mode but it was not able to limit its angle and speed! Something strange was happening. Even at Prearm you can see GPS and AHRS mismatch warnings!
@xfacta can you look into this please?
That is the craziest looking map I have ever seen
You can see the GPS path (blue) is where you expect it to be, and the IMU-calculated path is branching off all over the place
For the vibration levels, I believe anything over about 15 should be investigated or at least watched in case it changes over time and becomes a serious issue.
Anything over 20 is going bad and can sometimes even have serious effects even at that level - especially on Z axis.
30 is a no-go zone, you are already having issues.
Some IMUs handle those vibration levels better than others, so with your copter there is no clipping, but the position is clearly affected.
GPS Glitches happen when IMU-calculated path or direction deviates from the GPS path, so this doesnt automatically mean the GPS is wrong, but the flight controller could not tell which was correct.
I believe you should try to do all you can to minimise vibrations. If you have a lot of harmonics you might need a stiffer frame.
Unrelated to vibrations, but maybe check all connectors just to be sure, but I couldnt see where any signals or voltages disappeared.
The copter definitely seems underpowered - do you have a payload onboard? Anything you can do to reduce weight would be helpful.
As Dave suggests, update to latest stable firmware - there are some very important fixes.
If you get the repairs done and this is going to be flyable again, I would probably consider resetting to defaults, and start over with all calibrations and run through these things in the section below.
Definitely do this whole section even if you dont reset to defaults and recalibrate.
Battery monitor calibrated and at least reading an accurate voltage
Also check and set
INS_ACCEL_FILTER,10
INS_LOG_BAT_MASK,1
INS_LOG_BAT_OPT,4
If you do reset to defaults and start over, do not be tempted to just put back every setting you had before - only change what is required for tuning and safety using the above process and the standard tuning guides.
Once tuning is done and flight is good, then start to customise more settings to suit your flights.
There’s probably a few other things I saw in parameters that I would do differently, so from here on is not an exhaustive list but just some of what I noted
I see you’ve changed the EKF failsafe threshold and action, so that’s a sure sign there are ongoing issues
ANGLE_MAX,1500
You pitch and roll is already exceeding this just trying to fly.
I believe this is not enough to correct for disturbances such as wind, and you will probably need something more for ordinary attitude control too. I would use 3000.
I can understand LOIT_ANG_MAX being 15 , but be aware you may have to quickly change to AltHold or Stabilise if disturbances can not be handled during a flight.
INS_HNTCH_HMNCS,11 is probably a bad sign - trying to filter so many harmonics.
Once repairs are done just do a short test flight in AltHold with some pitch and roll.
Let’s see that log.
EDIT
Just wanted to add that I was mostly trying to address your vibration queries - as Allister pointed out it looks like something was coming loose and causing more vibrations that normal, along with the thrust loss message and Motor2 was being commanded to maximum.
………….
You are wrong. Your craft was setup to fail.
Yes I think that was being conservative.
There are cases were we can see Z axis vibrations go above 20 the copter would start rising, each and every time. So there is no doubt a few other factors involved, but if you have Z axis vibrations up around 20 then you are asking for trouble.
It looks like it’s related to terrain altitude, it’s not related GPS or barometer altitude.
But when you arm and raise throttle to take off, the altitude resets.
I’m wondering if this is all related to the IMU issues you are seeing.
I’d be seriously looking at total reset and recalibration plus the procedure I outlined and see if all this goes away.
Definitely do the firmware update before anything else!
@xfacta @dkemxr , Thaaaaanks aloot for your suggestions and time…
I would like to add few things, as explained by @Aorish_Imtiaz , this bird had several hours of smooth flight. Even at the same day when this event happened, the copter had smooth flight. Moreover, z-axis vibrations were under control and copter was able to mantain a very good altitude. We produced these systems for our customer/user after several trials and delivered them one year back in 2022, therefore u cud see the old firmware…
Regarding this crash, if you look at the log you wud see that the pilot tried to arm the sys but it was rejected by autopilot twice. Therefore, we are assuming that ekf estimates (from imu,baro and gps) were not valid and therefore arming cud not be done. Once the copter was armed, the estimates during flight were again poor but they might not be purely because of high z-axis vibrations. Could you see a chance that autopilot was ripped-off from the base or it is related to wrong imu calculations (may be due to invalid calibrations).
I hope u can understand that if there are several smooth flight just few hours before and suddenly u found a crash where the copter turned into wreckage (that cud not be repaired), it is very very important to analyze what caused the crashed.
Once again thanks for sparing ur time for log analysis…
Regards
Yes I read those other posts too, thanks for all the additional info.
I would power up that Cube and GPS and any other accessories that still work and see if you get the same IMU variations, the do the firmware upgrade and see if it goes away. Leave it all powered on for quite some time.
You could even have a faulty Cube.
Is it possible the autopilot didn’t initialize properly?
@rmackay9 @Leonardthall if you could give your expert opinion on this?
@priseborough you being an EKF expert, can you figure out what is happening with EKF outputs here?
Sorry @Aorish_Imtiaz, I can’t see what has caused the EKF to get so confused here. Vibrations aren’t great but not crazy, the two mag’s are a little different but it isn’t unusual for the internal one to be off. You seem to have a large velocity estimation, maybe bad accelerometer offsets.
Sorry but I don’t have the depth of knowledge to help you here.
