Does anyone have in insight on this flight? The drone began gaining altitude uncontrollably. I noticed the Compass offsets were very high. Could this have caused the problem? I think it also hit radio failsafe but RTL did not bring it back.
Setup:
Pixhawk 2.1
Large x8 rotor
Here+ GPS1
Here GPS2
LIDAR Lite v3 via I2C
Every other flight before this went well. This is a customers aircraft so I’ll work on getting more data regarding the flight conditions, weather, etc.
Copter starts flying an Auto mission at low altitude
Copter uses rangefinder to keep altitude, because EK2_ALT_SOURCE = 1
Transmitter failsafe triggers RTL
Copter begins to climb to RTL altitude, but the rangefinder jumps from 13 meters to 1.5 meters and stays there.
Rangefinder stays at 1.5m (or 0) as the copter climbs really high. EKF starts panicking because its primary altitude source has failed. It measures a height increase from other sensors, but resets back to 1.5m when it diverges too far from the rangefinder reading.
Land and Loiter modes are used to command descent. EKF is still freaking out for the same reason, except now it’s descending instead of ascending. Luckily, the EKF resets allow the desired altitude to be commanded lower than current altitude, so it descends.
Other things I noticed (that didn’t cause this problem):
Your rangefinder settings are… optimistic. Max range is set to 200m, and EK2_RNG_USE_HGT is 50, which means it will try to ONLY use the rangefinder all the way up to 100 meters altitude! I wouldn’t trust a lidar as primary source that high up.
You’re getting Z vibe clipping in flight. Check that there’s nothing banging against the flight controller such as loose wires and consider changing the dampers.
Baro alt seems unusually inaccurate while flying at low altitude. Is the flight controller exposed to prop wash or flowing air in flight? Consider enclosing it (not totally airtight!).
Recommendations:
Check the rangefinder, maybe something was blocking it. The 1.5 meter reading is erroneous.
Fix the barometer and set EK2_ALT_SOURCE = 0, and EK2_RNG_USE_HGT or RNGFND_MAX_CM to lower values. This will make the flight controller switch to rangefinder as primary source when in range, but will use barometer at higher altitudes or if the rangefinder craps out. I am assuming that low altitude flying and the rangefinder is important to your use case, if not, consider using the barometer more.
Consider adding a non altitude-controlled mode such as Stabilize, if you trust your customers to fly in this mode. Stabilize does not require the baro or rangefinder, so it will still act predictably in case those fail.
It’s pretty obvious we need to work on the LIDAR settings. I’ll look into what may have caused the erroneous reading.
Could light water or spray cause the LIDAR to stop recording correct altitude (This drone is a crop sprayer)? Or it is more likely related to a power issue?
Either way, I really appreciate your help! Interpreting the flight data is something I need to get more experience with.
Heavy rain, fog, mist, etc. does indeed foil lidars. However, I am not certain if that would show up as a consistent 1.5m reading. Definitely worth testing: climb up a bit and turn sprayers on/off while flying around and see if you get bad readings.
Actually might be able to do that with this log - there was a point where the lidar started reading 0m (no reading) instead of 1.5 meters, that could have been the sprayers turning off. Are your sprayers controlled by an RC channel?
I’m not familiar with LidarLites, so I’m not sure if this is a signature of power-related issues.
I made some changes to the rangefinder pages in the wiki that may be related: basically if the rangefinder max alt param is greater than the actual max useful range of the sensor, it causes problems.
This is a classic symptom of this problem - Z vibes can easily cause uncontrolled acceleration upwards. Could well be rangefinder problems as per @Anubis excellent analysis, but regardless make absolutely sure to fix your Z clipping.
I would like some outsider input at to what happened in this flight. The user indicated a loss of control an unintended climb similar to the above scenario. However, he was not equipped with a LIDAR.
We believe the compass calibration was performed incorrectly, however I also see excessive vibration.
The drone’s GPSes are intermittent. They frequently cut out.
They worked before takeoff, and the user flew in Stabilize for the majority of the flight, so it was never noticed that the GPSes didn’t work.
User entered Loiter while the GPS happened to be working, then entered Land mode. Throttle was left at 1630 PWM, because it doesn’t affect Land mode.
GPS cuts out, and the drone falls back into Stabilize mode. With 1630 PWM throttle, the drone shoots upwards (it hovers at 1250).
User brings the drone down in Stabilize
If the user did not hear (or didn’t recognize) the audible beep that indicates a flight mode failure, the sudden switch to Stabilize definitely would have felt like an uncontrolled event.
Luckily, the log also provided the answer to why the GPSes were failing: they drew too much power. Look here:
The top graph is the flight controller board voltage. The red line is the “power status flag.” It’s a bitmask that contains information about the board’s power management. Its outputs are defined here:
It starts at 3 (1 + 2 = power valid; good), but quickly jumps to 35, with occasional spikes to 43. This means the status is 1 + 2 + 8 + 32, which indicates that the power status has changed since boot, due to peripheral overcurrent. The second graph shows how the GPS status correlates to the the flag = 43, although the GPSes intermittently cut out more often than the overcurrent state is recorded. Towards the end, it looks like it finally gives up and permanently goes into overcurrent, shutting down both GPSes. The board voltage is also noticeably affected by whether the GPSes are in use, although it is not in danger of going too low (above 5V is good).
So the problem is certainly the GPSes drawing too much power from your flight controller’s serial outputs. There are solutions to this, but they depend on what hardware you’re using. What flight controller is this, and what other peripherals are connected?
Excellent analysis!
Do you think the autopilot switched to stabilize mode because of the GPS failure?
BTW, It is using the Pixhawk 2.1 and Here + GPS.
We have recently had a few Pixhawks show up out of the box with issues on the GPS1 port. One was intermittent and the other had a very loose connector. I will inspect the board closely to see if that may have caused the issues.
I’m hoping it’s not, we rely on these systems and would hate to find another failure
especially since we have great success with them the majority of the time.
Yes. If GPS (or EKF position in general) fails in a GPS-enabled flight mode, it will fall back to a non-GPS flight mode.
The Pixhawk 2.1 has two independent power sources for peripherals. There is a 1.5A rail that is dedicated to the Telem1 port (it’s meant for a radio, but you can put any serial device on it), plus a 500mA rail that serves all other peripherals. If you have a peripheral that draws a lot of current, put it on the Telem1 port. Something is sucking up all the power on the shared peripherals power line.
The specific problem in this case was clearly explained in the discussion here. Often uncontrolled ascent is because of vibrations, but it wasn’t this time.
