Typhoon Relief Mission X8 lost due to RTL Fly-Away

I just returned from the Philippines where I have been helping a local team to build and fly a mapping UAV to assist with the ongoing typhoon relief effort there. We built an X8 which was tuned and tested successfully yesterday. Today it had an unexplained fly-away.

Specs:
Skywalker X8 w/ 2x 5000mAh 4S
APM 2.5.2 using 2.76
Airspeed Sensor
3DR Ublox GPS
RDF900 telemetry on ground and in plane
1280Mhz 800mW FPV with 8db patch and Inverted Vee
Frsky Taranis with DragonLink
Mission Planner 1.2.93

Summary:
The plane completed its mission at 350m and entered RTL mode. It descended to its RTL alt of 150m and started loitering over the Home location as depicted on MP. However no one on the team could hear it or see it. FPV reception was very poor, and the terrain was not easy to make out. The pilot tried to switch to manual, but the apm remained in RTL. They tried to use MP to direct the airplane, but it did not respond. The telemetry link became weaker and weaker until it finally froze up.

Environmental:
They are located on the east bank of the Aklan RIver just outside of Kalibo. The mission was approx 7km from the airport (with ATC approval). There is a high voltage line about 1km to the north. The winds at 350 meters for most of the flight was out of the west at 5-7m/s for the duration of the flight. Clear skies with no rain, surrounded by mostly jungle with patches of agriculture.

Analysis:
My first thought was a GPS issue, but the tlogs show 11-14 satellites and an HDOP of 1.4 for the whole flight. [I can’t find the HDOP graphing option from tlog on MP, if someone knows please teach me].
It appears that the APM thought it was over home, but it was really somewhere else. If HOME was somehow set wrong, it wouldn’t show it circling overhead. The only thing I can think of is a large GPS offset that was not reported by the UBLOX.

I loaded the tlog into Droneshare, and it reported that the parameter CMD_INDEX was set to 0, outside of its range of 1-255. I dont know what this parameter is for or if it is relevant.
I’m at a loss as to how to proceed with this analysis.

This is a humanitarian mission, so please provide any help you can to get up back on track here.

Thank you very much.

Just as I was finishing up the Topic, I received word that the plane was retrieved. The police received a call that there was a plane crash, which of course created quite a scramble. The crash site was 10km away from the launch site. [attachment=1]crash site.JPG[/attachment]
The damage was minimal, just a broken winglet.
The problem is that the team has no confidence to fly it until there is a plausible explanation as to what occurred.

Another interesting failure. The Camera only triggered 7 shots, 6 immediately after takeoff while in manual mode, prior to reaching the first waypoint, and one more while it laid at the crash site. They were using DO_SET_CAM_TRIGG_DIST. Could this be related to a GPS fault? I don’t see how, but I’m at a loss for theories.
[attachment=0]flt pln.JPG[/attachment]

hdop is “eph gps_raw_int_t”

I’d say something happened to your GPS,

At 03m35s (tlog time(?) your ‘GPS alt’ disappears, as do the long / lat values, eph and ‘satellites visible’, coincidentally almost exactly as you start the mission.

The remote rssi (remrssi) is also pretty low after take off and eventually goes to 0 presumably as it flew further and further away.

The photo taking didn’t work because ‘distance’ was unknown.

The CMD_INDEX is inconsequential, I get it too and it’s not something you’re manually supposed to change.

Thank you Graham. Isn’t there a GPS failsafe now, or is that just for copter?
Are there supposed to be warnings when GPS malfunctions? Why did MP report a good HDOP?

From my experience with my own problems it seems the MP reports the last data received and continues to do so unless new data arrives. If you have a total loss of comms then it’ll report that but not the loss of a single parameter stream. But why it didn’t report Bad GPS Pos I don’t know. Maybe Michael or Tridge could tell us.

I think the GPS failsafe causes a circle rather than a Loiter or RTL which maybe what happened here as the roll attitude during the last part of the log is a bit strange.

Do you have the dataflash log?

No data flash logs. They were overwritten in the hours that the plane lay at the crash site. Perhaps we need crash detection to stop the logs and the prop.

The mode showed RTL the whole time, it never entered circle mode. Do if there is a GPS failsafe, it never entered it.

Checking the CLI shows waypoint 0 is the crash site. This is probably because it regained a gps lock after the crash, which re-assigned home. That’s when the last photo was taken, triggered by the distance change as the gps location focused in.

It happened again today!
They swapped out the GPS and set up a small rectangular auto mission. The plane reported following it, but it flew away and they lost it again. They certainly chose too high of an altitude for a test like this, however despite several choices I would have made differently regarding conducting this test, the fact remains that that GPS was lost with out warning or notification on mission planner. The HDOP and number of sats showed good for the whole flight, but looking at the tlog, all GPS related info stopped just after takeoff throttle is set.
[attachment=1]GPS stops at take-off2.JPG[/attachment]

What is going on here!?

  1. What is causing this failure? It must be the APM since the GPS was swapped. Or maybe a bad cable that is affected by vibrations?
  2. Is there an error in MP that it doesn’t report this failure?

Michael O,
We have had a couple fly aways caused caused by a mysterious GPS issue. Both times the operator was not aware of the problem because it was not alerted in MP. Looking back at the tlog shows a total loss of all GPS failure just after takeoff, but MP continued to show a high sat count and low HDOP.
The whole story with the logs is here:
viewtopic.php?f=44&t=5808&p=9804#p9804

[Stefan, I’m sorry for the cross posting, but this issue really does fall into two categories and I think Michael should be in the loop. Thank you for you understanding]

what baud and air rate where you running the radios at?

for some reason GPS_RAW_INT stops being sent, this message contains gps hdop, sat count, groundspeed, ground course. the ground course you can see is the black line when you play back the log, and it just sticks in one direction.

I do not know the cause of why this would happen however.

hi I had very similar problem quad flyaway for no reason studying logs showed gps disappeared for a split second turned out to be vibration on conector to gps bit of hot glue gun on wires and conector cured it not had the problem since
hope this helps
stu

the only thing I can assume is that it was dead reckoning, and lost the gps right after takeoff.

ok looking at the code

static uint32_t last_send_time;
if (last_send_time != 0 && last_send_time == g_gps->last_fix_time && g_gps->status() >= GPS::GPS_OK_FIX_3D) {
    return;
}

if the apm hasn’t received a new gps message, it will not send a new position to the gcs. So this looks like its a connection issue between the apm and the gps.

the more I look, this looks like a apm failure, not reporting that its lost valid gps lock.

That board will be destroyed, never to fly again!
Is there something you can do to help MP identify such problems in the future? If telemetry no longer shows a stream of valid gps data, even if the APM isn’t reporting a failure, MP has the information it needs to report a possible failure to the pilot.

Not sure off hand, but it was all default settings.

[quote=“meee1”]ok looking at the code

static uint32_t last_send_time;
if (last_send_time != 0 && last_send_time == g_gps->last_fix_time && g_gps->status() >= GPS::GPS_OK_FIX_3D) {
    return;
}

if the apm hasn’t received a new gps message, it will not send a new position to the gcs. So this looks like its a connection issue between the apm and the gps.[/quote]

If no new GPS position was sent from the APM, then why did the GCS show the plane dutifully loitering around home when in reality is was 10km away?

The X8 was retrieved again, thanks to the help of the local villagers. [attachment=0]ImageUploadedByTapatalk1390741526.308041.jpg[/attachment]

This time the damage was more substantial. 1 wing was destroyed and the spar broken.

The data flash log is here: dropbox.com/s/qzmzjuan622u6 … yaway2.log

Here is the FPV video. You can see that the GPS coordinates are locked up which went unnoticed by the pilot.
drive.google.com/a/hawaii.edu/f … sp=sharing

Michael,
Should I open a github issue for this?
Thank you.

Hi Iskess,
Sorry for your fly-away!
The short version of what happened is you lost GPS lock, and unfortunately a bug in the 2.76 firmware meant that the loss of GPS lock was not reported by the ground station. That bug is fixed in 2.77.
The longer version is that for 2.76 I tried to improve MAVLink performance by not sending some messages when there was no new information to send. For example, it avoided sending new GPS_RAW_INT messages when you don’t have a new GPS fix. The bug was that it also avoided sending the messages when you had no GPS lock, so when you lost GPS lock it didn’t send any more GPS_RAW_INT messages.
The fix for this was to always output GPS_RAW_INT messages if you don’t have GPS lock.
When you lost GPS lock the APM automatically went into dead-reckoning mode, and kept flying the mission. The dead-reckoning code isn’t accurate enough for really long missions, so it was way off course by the end of the mission.
A second problem is the ‘health status’ bits in the SYS_STATUS message. There is a bit that indicates if the GPS is attached, and a 2nd bit that indicates if the GPS is healthy. Both bits stayed at 1 as the code considers the GPS to be healthy if the GPS is responding and sending messages. It doesn’t set the GPS as unhealthy if it loses GPS lock. I’ve changed that now so that the SYS_STATUS message will show an unhealthy GPS when it loses GPS lock.
There are two more aspects to your log I think we should look at. First off, why is it losing GPS lock? It looks like RF interference of some kind as it loses lock in both logs as soon as you start flying. What electronic devices do you have on the aircraft? Do you have an analog FPV transmitter for example? What types of telemetry radio do you have? Does your RC receiver have telemetry built in?
The second thing I think we should look at is if we should add a GPS failsafe option. Right now it relies on the GCS operator taking some appropriate action when you lose GPS lock. The lack of messages on the GCS when you lose lock was terrible of course, but even with that now fixed there is a problem that the GCS may not have contact with the aircraft at all times. I think we need to add a FS_GPS_ENABLE option that allows you to specify the action to take on losing GPS. I think this should behave much like FS_GCS_ENABL, and allow you to first start circling at FS_SHORT_TIMEOUT then to RTL at FS_LONG_TIMEOUT. The RTL will be via dead-reckoning of course, but straight line dead-reckoning is usually OK as long as you have a compass. It should at least move closer to the return point than it would otherwise.
Cheers, Tridge