Started flight testing an orange cube today on 4.0.1 with a small hex frame. Setup reportedly went okay. However the first flight was erratic.
The copter flew alright in stabilize with stock PIDs. After switching to altitude hold the copter was erratically changing altitude and drifting in height. The EKF health was very poor with velocity variance being the main culprit. Oddly enough the EKF vertical health was not particularly high despite the erratic altitude changes.
Just to witness the behavior, we also switched the flight controller into loiter mode. The copter drifted at about ~2m/s for a few seconds before entering EKF failsafe and landing.
I have uploaded a quick video demonstrating the behavior as well as the dataflash logs for the flight. The video was recorded during the second flight in the log file. We are going to look closely at the problem tomorrow. We will probably check over the aircraft again then re-calibrate the sensors. This copter did have a Here Flow module connected over UAVCAN. However, Flow_type is set to 0 to disable it for now.
If anyone has any time and can take a look at the log and video it would be appreciated. Also, thank you to all the devs and maintainers for the hard work. Looking forward to putting all the new features and improvements in 4.0 to use.
Hey there. This is an easy one to diagnose - the vibration levels are regularly over 40m/s/s and spike as high as 90m/s/s. As a guideline, below 30m/s/s is good, between 30 and 60m/s/s is the grey zone where altitude estimation (and thus control) issues may occur, above 60m/s/s is where you’re nearly guaranteed to have problems.
Thank you @rmackay9. Glad you were able to spot the problem quickly. I never knew what the units were for the vibrations level so that is very informative as well.
We are going to change the props to a stiffer type, make sure no wires or components are loose, double check the mounting on the carrier board, re-calibrate the sensors with extra care, and fly again. We’ll report back soon. Thanks again!
We checked everything out and did another test flight. Vibrations were still high, but that doesn’t appear to be the issue anymore. On a hunch we flew without the UAVCAN GPS connected. The difference was night and day.
The EKF becomes unhealthy shortly after takeoff with the GPS connected. With the GPS unplugged, the copter holds altitude as expected and the EKF returns to good health. I have attached a video and logs including back-to-back flights with and without the GPS attached.
That GPS unit, and indeed the whole copter, have previously flown before this testing. The only change made was swapping the black cube for an orange cube and the corresponding firmware update to 4.x. We will test connecting the GPS unit over serial to see if that is a factor. If the problem persists we will swap out the HERE unit. We may also try swapping the black cube back into the drone then doing a test flight (mostly to characterize the vibrations).
If there’s anything else I should try or test out, let me know
In the bad log it looks like it is constantly switching EKFs and this is probably caused by very bad interference on the compass from the motors and powerwires. Below is a graph of the compass field strength (in red) vs the throttle output (in blue) and we can see a very clear correlation.
It’s important to get the compass up and away from interference.
By the way, the two problems you’ve hit so far are very common issues that users hit on new builds. Not trying to put you on the spot or anything, we’ve all been through this.
Whelp, we found the problem…This one is an embarrassing and rather funny mistake. The GPS position offsets were accidentally set to tens of meters. Any rotation in the aircraft, even when not armed, would cause a significant velocity variance as a result. Problem solved and we’re in the air on an orange cube with 4.0.1!
As an aside, we are using the HERE unit over CAN and as a result the external compass is mag2 and the internal is mag1. We have the primary compass set accordingly. We do still have high vibrations causing problems with altitude holding which we will have to reduce.
Edit: Would you recommend disabling the internal compass on any aircraft where it has high interference?
Thank you for your time, help, and expedient responses. Looking forward to testing more on 4.x and enjoying the improvements you have all worked so hard to create.
Aha! Yes, indeed, I totally didn’t notice while looking at the params that the GPS_POS1_X/Y were set to be very large. I wonder if we should reduce the (soft) limits on these parameters to ±5m instead of the ±10m they are now. It’s hard to imagine we have many vehicles that are more than 5m across and this might reduce the cases where users accidentally think the parameter scaling is cm.
Yes, I probably would recommend setting the COMPASS_USEx param to zero for compasses that suffer from a lot of interference. The only value those compasses add is a pre-arm check that the external compass is oriented correctly. This check is useful but I suspect that the danger of the EKF using the internal compass when it shouldn’t is perhaps worse.
