I really hope someone with more experience than myself can help me debugging my last flight, which ended in a sudden crash after successfully flying for many many months using an APM and now the Pixhawk (which had around 5 flights).
After flying around for some minutes in acro mode the copter suddenly turned exactly upside down. Not just rotating around like when a prop/motor came off or so. It felt through the glasses like the copter thought, that its current position of the copter should be bottom-up! I only had a few seconds to try to recover while still having visual image on the fatshark, so I remember I first tried to switch to stabilized with no luck, then back to ACRO to turn it over again. And finally I hoped the RTL switch may save it while losing visuals.
Nothing of it helped and the copter went straight down, luckily into a field/grass by only breaking the GPS Tower, an arm and the top plate, leaving most of it intact (including the GPS tracking with wich I could locate the copter)
To help debugging the issue it may be good to know:
I carried a Mobius action cam instead of my GoPro the first time
I switched to the extended kalman filter for the first time before this flight
Version: 3.2 RC5
Many many thanks to anyone in advance taking some time to look into my logs!
Uploading the log to droneshare said the same. I’m not so sure about that as the exact problem (maybe that was end of the log was when the real crash happened on the ground?!) because:
the place where it went down was not at the same height as the one where I started, so there might be a difference
I remember many flights where the copter logs said that I was on height X when I switched it off, while I was sure that I cut off the battery right at the same place where I started. So I often thought that the height calculation in the logs may be wrong somehow…
I had two redundant power sources connected. One BEC on the main input (still need to check that one here on the table now) and one of the BECs on the ESCs was connected too. Also I had a perfect image from the fpv equipment with the minim-OSD showing enough battery voltage and only ~3000 of 4000 mAh drained or so?! Wouldn’t the Checking the battery afterwards also showed around 25-30% capacity left.
Hmm…Can you say what kind of power failure? As I said, I still had the minimOSD up and running too and it felt that I still had the motors running, although the whole copter felt like it tried maintain its level upside down.
mapcoordinates.net says it was 31m below the takeoff point. So the log and/or the power cut off in the air?! Still it feels like that may have happened a little/second later after the initial 180 degree roll-over but for sure could be related somehow as the 3D flight screenshot above shows too.
Maybe it burned out because of some full speed on the motors + gravity making the motors spin way to fast while falling upside down?! I may sound stupid here, just guessing for sure… sry!
Is there any hint about the redundant power setup or why that may have failed in this case, since the copter had enough battery.
Checked the BEC. Still works at 5.34V. Same with the ESC power cord. Nothing burned out so far.
The 5V main BEC can handle 3.5A and drove the Pixhawk, GPS+
Compass, 3DR Radio, MinimOSD and the receiver. So not really anything which would bringt this up to 3.5 amps.
The FPV transmitter and cam had their own BEC.
Re redundant power supply: looks like it was ready and able to switch according to the flags.[/quote]
Not sure if I understand correctly. Does this mean that it was ready to switch, but didn’t see the need to do so because the main BEC was still operating fine? If so, the pixhawk may itself have burned out?
Side note to that: I still need to check the FC itself. Until now I only took out the SD card and had the pixhawk only connected to the battery for some seconds with the LEDs coming on. So I still need to check if its running at all.
[quote=“jschall”]Wait, do you have a telemetry log?
What do you mean by GPS tracking in your original post?[/quote]
I’m running a 3DR Radio most of the time but I’m not sure if my laptop was on for this flight (I may check this later). Anyway the distance was way to far for having live telemetry, since it normally cuts off at max 500m or so (I was at 1000m away at the time of the crash).
For the GPS tracking: the copter always carries a GSM tracker on the top which saved the whole copter this day. I just needed to send an SMS to the device, which answered which almost exact coordinates to find it (glad that it didn’t go down on concrete g).
Also I connected the USB cable now. No chance when trying to connect through the APM Planner itself. BUT: trying through the Terminal connection I instantly get connected to the “NuttShell” on both, 57k and 115k ?! oO What does this mean?
[quote=“DerFlash”]
Also I connected the USB cable now. No chance when trying to connect through the APM Planner itself. BUT: trying through the Terminal connection I instantly get connected to the “NuttShell” on both, 57k and 115k ?! oO What does this mean?[/quote]
Ok. After reinserting the SD card, everything runs fine again. Connected without any problem, Pixhawk still seems to work fine!
Still unsure what exactly was the problem. So if there’s any more hint about it, please please let me know. It would be really bad to just “hope” that it would not happen again, since I had a whole lot of luck that it didn’t break everything.
Anyway, big thanks to jschall for looking into this so far!
I tried to find out some more information by using the APM planner and came across some interesting things:
At first I noted a little spike in HDOP almost exactly when the altitude finally went down shortly before the end of the graph. It from 1.29 it went up to 1.4 and at the same time the satellite count dropped from 10 to 9. Not really bad I would say, but still interesting that it happened exactly at this time frame as you can see in the attached image.
Also MagY and MagZ spiked at this time - maybe a hint on the “flip to upside down” I spoke about in the first post?!
Keep in mind that this seems to be the moment where it started to drop upside down. That’s the interesting and questionable action and I need to find out why it did that.
I would be glad if any developer could take another look at the graphs and/or log and may read it better than me to find out the reason for the flip.
[attachment=0]Bildschirmfoto 2014-10-08 um 21.03.40.png[/attachment]
Also in the screenshot you can see that the log seems to continue counting the lines while though writing ERR on most of the graphs. This was the reason I thought I should export it to a txt log file and look into it using a text editor.
And there I found some more interesting things:
The very last log line says “EV, 18” which means “DATA_LAND_COMPLETE” oO
Why the hell would it try to land during flight? So I tried to find a previous line with “EV” and found it almost at the top of the file on line 1055/54203 as “EV, 28”. This one is defined as “DATA_NOT_LANDED”.
So something is really wrong here…
After googling about the last one I found this post on diydrones: diydrones.com/forum/topics/any-i … not-landed
Seems like I’m not the only one with this happening and I ran into a bug in the firmware here?!
Again, keep in mind that this was my first flight with the EKF enabled and carrying a mobius action cam which is said to create a lot of “noise” on some GPSs/MAGs?!
(sorry, I can no longer edit my post, so I need to add another one here)
Ok, reading the code, the “EV, 28” just means that the firmware logged that we “took off”. But that still does not explain why the last line states a “land” while the copter flipped in the air and crashed. Somehow the calculation seems wrong and it may have thought that it landed - cutting off the power on the motors?!
Ah, I should’ve caught that before. You’re running 3.2-rc5 (we’re up to rc11 now) and the land detector was not strict enough in that version. You had an in-air disarm.
Sorry about that, 3.2 was and still is experimental code, though we are getting very very close to releasing it.
[quote=“jschall”]Ah, I should’ve caught that before. You’re running 3.2-rc5 (we’re up to rc11 now) and the land detector was not strict enough in that version. You had an in-air disarm.
Sorry about that, 3.2 was and still is experimental code, though we are getting very very close to releasing it.[/quote]
Ah nice. No need to apologize. For sure I’m fully aware about the fact that I wasn’t using a stable release and that this means a higher risk for problems like this (although “release candidate” versions should not result in such a crash, but shit happens ). But trying to help debugging release candidates is more fun and most of the time the issues are less problematic.
I’m glad that I finally know what went wrong, so big thanks for your time, jschall
PS: Now that you spotted the reason I remember doing one ore two manual loops in ACRO just some second(s) before the bad flip happened.
PPS: One interesting thing though about this, which someone should maybe check: The “wrong” reported height. Since it dropped 100-200m or even more, the log seems wrong about this.
[quote=“jschall”]It stops logging once it disarms. Maybe we should add a delay to that so we can see if it falls after it disarms.
[/quote]
Ah, I see. Yes, logging a little longer, could help to debug such situations.
One last thought about this:
Since I’m flying a lot in ACRO mode, I’m still a little concerned about the land detection, although it is more strict now. Assume we could fly upside down (by having reverse operational motors), could it make sense to ignore the land detection in some situations or even whole flight modes like ACRO?!
Anyway, thanks again for your help and work. Please keep it up!
). But trying to help debugging release candidates is more fun and most of the time the issues are less problematic. 
