Log Crash Analysis Help

Hi All,

I have been flying a 3DR Solo and on my last flight I sent a mission to the drone and looked back up and it dropped out of the sky. This was only a few minutes into the flight and on a full battery. I am still pretty new to Mission Planner but didn’t see anything stand out on what may have caused the crash.

I posted this on the 3DR forums as well and have only gotten one response. Currently the thought is that the channel 3 motor pod may have failed based upon the RCIN vs RCOUT signal. It was a fairly windy day (20 mph winds) but I was trying to keep the travel speed under 15mph. Does anyone have time to take a quick look at the logs?

https://www.dropbox.com/s/a04po9r7hxczypl/2018-11-22%2015-50-21.zip?dl=0

Just trying to determine the root cause so I can avoid another crash in the future. Thank You!

1 Like

Your left rear motor failed to provide thrust.
You were flying Solo Firmware 1.3.1 - very old, dangerous ESC signal voltage level bug bit you.
Make sure to update to 1.5.4 at least. https://github.com/OpenSolo/ardupilot-solo/releases
( or current ArduCopter build - if you replace the cube, or modify your own with madhacker.org 5v mod)

The easiest way to upgrade to proper version is to run Solex (Android app) , and ask it to install OpenSolo (which fixes everything for you.)

1 Like

Thanks Andre. Can you expand a little more on how it failed to provide thrust? Was this a mechanical issue with the motor pod or are you saying the failure was caused by the firmware? Is the RCIN vs RCOUT graph what you used to determine this? I have another solo that I plan to move to and will definitely upgrade to the 1.5.4 firmware before flying next time.

Fingers crossed, @Pedals2Paddles will have had enough eyes/hands on the upcoming OpenSolo on Master build by then that you’ll have the latest and greatest base firmware from ArduCopter, along with the new slew rate limiting code, which was initially designed by 3DR to prevent the ESC brown-out. Matt and others have been working on getting that code updated, improved, and mainlined for the last half-year or so.

Yes, Slew rate were earlier deemed at too dirty solution, but may spawn in master, making all Solo’s happy.
@Brian : RCIN is RCIN , for example, RCIN 3 is throttle input, while RCOUT 3 is output to one of the ESC’s , you can 't compare those like that.

The explanation requires some electronics talk, Ill try to make it short;
The Solo ESC MCU works on 5v , the solo Cube 2.0 provides 3v3 signals , which is fine, as long too big load does not cause the ESC’s ground potential to go up due to internal resistance in the GND plane/wires.
Once it does, the ESC fails to “see” the high periods of the PWM as “high” , and thinks it lost signal, - it stops , then the current drops, and it sees high throttle demand, but refuses to start on high throttle (safety feature)
3DR solved it by adding a throttle slew rate, limiting current spikes.
This code is needed in order to fly a 3DR solo reliably , once you have a cube mod, a upgraded firmware, or new cube(kind of extreme solution) , the Solo is very reliable.
I recommend OpenSolo & Solex , a great combination. ArduCopter once it have slew rate, or stock, 5v modified cube with ArduCopter.

Log is somewhat inconclusive, which itself is an indicator. First, there was about a 2 second gap in the log data before it crashed. Then at the very end, the copter appears to be tumbling. The 5v power to the cube sharply drops, and the log abruptly ends. There is no change to auto mode in the log. And no other meaningful information I can see.

I think you had a power failure, or something along those lines. It doesn’t appears to be any specific motor failure, or anything you did wrong.

There is nothing unsafe about 1.3.1. That’s what every unmodified solo is flying and has been for years. Upgrading to 1.5.4 doesn’t functionally change anything related to the ESC issue, and both have the same protection in place. I do not see any motor loss in these logs.

1 Like

Thanks for looking at this Matt. If you are leaning towards a power failure would that be possibly something within the circuit board or a bad battery or battery connection? All of the batteries I have been using were “new” and had less than 10 charges on them but I imagine they had been sitting on the shelf for some time. I am now wondering if this may have been avoided if I had checked the batteries with madhackers battery tool or solotool?

gap in log ? where ? - it must be late or something, I saw nothing like that

Near the end you see the long stretch of no data connected by a straight line?

@Andre-K I am still trying to decipher your summary. My electrical engineering skills are a bit lacking. Do you think a high gust of wind could have caused a high load demand, causing it to fail? I have seen the opensolo firmware mentioned quite a few times but have been hesitant on upgrading as I was afraid it might void the warranty for any solo’s purchased that were still under the 1 year warranty. This might not be true.

Is this the upgrade you are referring to:

  • Low battery thrust priority: If the battery is getting dangerously low, it will allow itself to sacrifice yaw (rotation) control to increase thrust to prevent a crash. This could give you the thrust you need to land softly rather than dropping out of the sky.

@Pedals2Paddles I am probably not looking at the right graph but when I plot Vcc and scroll to the end of the log the timestamp seems to line up with the last voltage reading. See below.

Looking at it a little closer, I agree it was a motor failure. The power loss was at the same time altitude reached zero. As in, the power failed due to the aircraft smashing to pieces on the ground.

Looking at the logs again, I realize now the gap in log data is actually during arming while on the ground. So nevermind that.

@Pedals2Paddles are you still thinking the drone would have crashed even if it was flashed with 1.5.4?

100%. The slew rate protection is there in both versions, and is unchanged in both versions. It’s likely your motor failure was not a result of that ESC bug at all, and just a random hardware failure that can happen once in a while to anyone.

I concur , as @Pedals2Paddles wrote before, the slew rate was before 1.3.1 - I just remembered 1.3.1 as a very old version, and jumped to the conclusion that it did not had the throttle slew rate.

So there was a failure, but not due to old firmware.
The company I work for bought up a bunch of used Solos of ebay, we have had lots of flights, never experienced a crash that was not en pilot error. A crash such as this would have us to check that ESC very closely, and, if possible, verify that all cables were still plugged in after crash, or the possibility for some DF13 or power connector to the ESC being partially connected.

@Andre-K and @Pedals2Paddles I was thinking about this more. Do you think it would be feasible to add in some sort of warning system into Solex that would monitor these motor pod parameters? Andre, I saw your graph above and not really sure which line to look at and which output is showing the failure. Maybe a motor pod failure happens to quickly to react to? If it is a slower/continuous indicator it might be nice to have a warning show up on the screen. I am guessing this would have already been incorporated but thought I might as well ask the question.

Monitor what parameters? Do you mean look at the actual pitch/roll vs desired pitch/roll? It’s too late by the time that becomes apparent. If a motor or ESC fails on a quad, it’s pretty much over.

I guess I was initially thinking in terms of graphing/monitoring load on the motor pods. Obviously this will vary quite a bit with weather conditions but could you have it spin up on the ground without takeoff and compare the loads/draw for each individual each motor pod. If one is drawing more than another it could possibly indicate it needs maintenance? Just brainstorming.

Solex / AP has no way of predicting these things.
Yes, you could do a thrust/power comparsion to detect some faults.
And: it is very uncommin, my job had 8 (used) Solos retrofitted with custom payload in 2018 , flew … a lot, no issues.