Nimbus 1800 VTOL Crash Analysis

GregCovey · September 9, 2020, 7:06pm

One thing I see different between the flight modes on the crash (first image) and the good flight (second image) is that the crash flight modes contain several QRTLs before AUTO. This typically happens when you are between mode selections on the 6-position knob. I have seen my flights get screwed up when not coming out of QRTL properly (software issue).

Your motor outputs look clean on the good flight transition.

As does the Altitude, Speed, and Current.

Let’s ask @hwurzburg Henry what he thinks and if there is a way to eliminate the in-between mode calls to FLTMODE6…which you have programmed for QRTL.

hwurzburg · September 9, 2020, 8:40pm

…the logs show everything fairly normal while it VTOL, transition wait, and things go wrong at transition done when VTOL motors are not stabilizing any more…but once flying the demanded pitch vs actual diverges rapidly…the motors cutoff since the TECs speed is WAY above the desired speed (25m/s vs 14m/s) as its diving to the ground…its trying to pull the nose up but it just goes further down…btw, the TECs speed range is very narrow 11 to 16m/s …cruise is 13m/s…pretty slow for a heavy bird…
given that this flow in fixed wing mode… it could be that it just stalled immediately once transition was done
as to the mode stuff, in the CRASH log, the modes obeyed the mode channel exactly (AUTO must have been initiated from GCS or switch…its not on the mode switch)…before AUTO takeoff, there was a lot of mode toggling, but that was irrelevant…

edited: I see that it has flown fixed wing succesfully…so my guess about control setup has to be wrong

Chad_Frazer · September 9, 2020, 11:35pm

Yeah, this bird has flown more than a few dozen times as a fixed wing.

Winds aloft started at 4mph on the ground and by the time it reached 150ft they were 14mph, during transition it dipped to 12mph. None of those wind conditions should have stalled it…

When it was on the way down I could tell from the ground that it was not normal behavior so from the RC controller I was pulling max up but of course, it did not seem to respond.

The speed being 55mph (25m/s) is consistent with other flights as during the transition she eats .5V out of a healthy battery. Being a twin engine she can get some speed going. It’s that speed buildup that makes me always put a do change speed command to 32mph, else she gets close to the overs peed condition in short order.

I don’t remember what she weighed but its around 10 pounds iirc. At 32mph she flies with a 15deg nose up angle.

On my RC transmitter I have a three position switch, QLoiter, FBWA and RTL that is on the right. On the left I have a two position switch that only activates QRTL mode. Since I fly in places that is crowed by trees and/or obstacles I find that letting it circle overhead until I am ready and then switch to FBWA, get her to fly in close by hand and then manually trigger the landing is the best way to handle her.

Yes, I can only institute AUTO mode from the GCS, I can’t justify bumping the transmitter into AUTO mode by accident. This also ensures that I have telemetry setup and running before it can go into AUTO mode as well.

NEW: So, what about this? During setup the left wing aileron was bumped by me and it stuck in a full down position. I thought I hit it hard enough to strip the servo. I manually put the aileron back into position, tested all flight control surfaces and it behaved perfectly.

During the post mortem I was curious to see if that could have been the cause. I dismantled the servo and inspected it. All gears were fine (perfect, had a small bit of oil on them still). By perfect I mean the gear teeth all meshed and none were broken/missing. No foul odors, nothing physically wrong with the circuit board. I then put the control horn back on and moved it to see if maybe the plastic side got its spline shredded. If fully engaged I could not make her skip splines. If partially disengaged but still attached to the servo she would indeed jump splines.

SO: Could the servo screw attaching the control horn to the servo have vibrated loose over time and the left aileron got stuck in an abnormal position, say a nose down position) and caused this crash that way??

EDIT: I just hooked the servos back up to the receiver and they appear to be just fine. No skipping, no abnormal noises and I can’t make them move while pushing them with my finger with a fair bit of effort. Control horn was fully engaged for this test.

GregCovey · September 10, 2020, 1:17pm

It did respond to as far as it could raise the elevator. Your MIXING_GAIN was only 0.5 which limits the amount of offset you can control on top of the automatic control from the flight controller. I typically set my MIXING_GAIN to 1.0 on most of my setups or I have to make these rather large turns around the field.

The UAV Log Viewer showed that your ailerons were functioning fine. You can also see that you did not fall from the sky so your pitch was slowing the descent as much as you (or the flight controller) could adjust the elevator. I too love this viewer!

Henry, thanks for looking. I am not a fan of having my 6-position switch on my Taranis and Horus make a mode change so fast in between switch positions. In fact, I make sure that my mode changes are fast and typically one-position at a time. I have seen bad things happen when changing modes to QRTL as if the software doesn’t clear everything that was initiated when you go back to FBWA or LOITER. I am not saying this is a definite bug but I am very careful now to only change to my desired mode. Perhaps a new setting can delay a mode change with a double check. Just a thought.

hwurzburg · September 10, 2020, 2:18pm

15deg nose up cruise? seems really high…close to stall…
…some searching around and I found the specs for Foxtechs turn-key Nimbus VTOL 1800mm:
Specifications:

Version: Nimbus VTOL V2
Wingspan: 1800mm
Length: 1300mm
Suggested Max Take-off Weight: 6kg
Suggested Payload: 800g
Total Weight: 3.36kg(no battery)
Max. Flying Height: 3500m
Max. Flying Speed: 35m/s
Average Speed: 18m/s to 19m/s
Stall Speed: 14m/s

dont think the Ail was involved…looked in control during the crash…

possibilities:

sustained stall…the pitch divergence began about 0.5s before the transition wait ended at 6:29.4 and VTOL stabilization stops and is flying at 12m/s (already above min speed of 11m/s, this needs to be changed I think)…stalls, due to reduced stabilization at the end of transition, nose drops, AP applies full up…stays in stall accelerating toward ground
not enough throw in Vtail pitch, like Greg suggests to keep nose up
CG shifted radically toward nose and it flew to the ground instead of stalling
Tilt motors not correctly aligned as they transitioned from 45deg to forward position exacerbating any of the above

sorry best guesses from the log

Chad_Frazer · September 11, 2020, 5:45pm

I’ve forwarded this problem to some older more experienced hands…I HATE a plane crash mystery. I’ll keep ya’ll informed as to the findings.

Karl_Schoelpple · September 15, 2020, 12:11pm

Hi Chad,
I’m an electrician / technician and also involved with drones for 9 years now.
If I may say your motor & ESC combination is not a good idea. I would strongly recommend having a much higher rated ESC in regards to current than your motor.
a) Motor current rating is based on certain test conditions. But in reality you may put more strain on a motor than expected. i.e a larger than appropriate propeller, or higher supply voltage than what was supposed to be ideal for motor.
b) Rush in current: This is often overlooked as it is mostly only known to electricians, electrical engineers, some electronic technicians dealing with motor controls …
During the start of an electric motor a much higher current is encountered within the first second or two. In fact it happens so quickly that the log (.bin or tlog) won’t be able to pick up or record. Only specialised test equipment will record this.
To give you an example a regular household 230-240V motor in your fridge or freezer (compressor) will take up to 10 times more current during start-up than during normal operation.
So in your case your front motors might in fact draw as much as 700A at the moment you start them. Needles to say the risk of a ESC then suffering damage if only rated at 50A is rather high. (Yes, I know they can briefly cope with a higher current but how much higher and for how long is the question here).
c) All electrical specs are usually based on 20C° …sometimes on 25C°. So the actual ability of an ESC to cope with a certain amount of current depends also on the surrounding temperatures or if it is already hot from a previous flight.

So best practice is to have an ESC which has a much higher current rating than the motor you will be using it with. In that way it won’t break a sweat even on a hot summers day.

Happy and safe flying.

Chad_Frazer · September 15, 2020, 9:21pm

I appreciate the input Karl!

This crashed bird was based on Foxtech’s system. For me it was a prototype. She did way better than any prototype should have and I’m going to build the next one. I’ve already got most of the parts in right now.

The gen2 bird will have 70A ESC’s. I normally like to have an ESC that can handle 20% more than what the motor’s continuous current rating is but in the prototype’s case I was dealing with so many new things that I wanted something that was proven to work, which was foxtech’s setup.

I can also say that the 50A ESC, when you read the manuals, is misleading. It’s 50A burst, not continuous! Its a 40A continuous ESC. It has some nice features but yeah, that’s a little low. It worked for as long as it did because the cruise amperage in flight is stupid low, like 5amps per motor.

So I can understand why Foxtech went that low but as you said, when in the field, things get hot. Texas (where the flight occurred) was hot and humid that day, like 103F with 70% humidity. There was a full ground run-up performed before the crash as well. So yes, I will definitely upgrade the ESCs!

I have a professional crash analyst taking a look at this one, I have the report already but I want to talk to him in person before I post the results.

Stay tuned!

Chad_Frazer · September 16, 2020, 10:41pm

Ok, I’ve had some time to piece this together. I had to call in an old drone buddy that is, well, very good at this stuff.

The official report:
Systemic Components Assessment

Autopilot Functioning Normally
Critical Flight Controls Functioning Normally
Pilot Input During Recovery Active

Contributing Causes

Demonstration Atmosphere
Preflight Inspection Missed FOD
FOD-induced Motor Surges
Transition Interruption Due to Motor Surges (FC algorithm needs work)
Pilot Failed to Execute Transition Reversal (may not have prevented UFIT)

Causal Factor

Aerodynamic Stall Resulting in Uncontrolled Flight Into Terrain (UFIT)

So, it is a combination of the FOD mixed with a low transition speed as well as PID limitations. It’s a mix of all of our theories.

The full story:

There was a previous attempt at a takeoff where the left landing gear had sunk into the ground due to heavy rains earlier that morning. This caused the aircraft to stick on the left side and flipped it over.
There was no structural damage but mud had embedded itself into the pitot tube as well as into the right side motor and both props were muddy.
The props were cleaned.
The right side motor had the mud removed and a spin check was conducted. The motor did snag on something. That something was thought to be mud that would clear itself on a full engine run-up.
The pitot tube was removed and cleaned, mud flakes were cleared from the plastic tubing.
The aircraft was prepped for another takeoff attempt during which an assistant blew hard into the pitot tube (possibly doing nothing, possibly damaging it).
As part of the flight prep I gently blew into the tube and witnessed the airspeed go from near zero up to 22mph and back down again which is normal behavior so I assumed it was fine.
I then held the bird down and let the motors go to max for about 10 seconds (not sure how long). This may have damaged an already under rated ESC.
An accelerometer calibration was performed and passed.
No errors were shown on the HUD before takeoff. All systems looked fine.
Auto mode was activated.
The aircraft struggled to maintain its GPS location by moving in a circular motion as shown by the autopilot .bin logs. From the ground it looked as if either the magnetometer was having an issue (no warning for a magnetometer was displayed) or it was fighting a moderate wind vortex created by being so close to a tall hangar’s corner where such a vortex could form.
Ground wind speed was approximately 10-12mph for this whole event.
The aircraft looked as if it recovered from instability once it cleared the hangar’s roof further enforcing the thought that it was a wind vortex problem. This was never confirmed.
Once the aircraft reached 150ft a transition was initiated.
As the props tilted forward it gained velocity. At 11 to 12m/s the autopilot reported transition speed reached but failed to achieve forward flight with sufficient power to avoid an aerodynamic stall.
All critical flight control surfaces were functioning as expected.
Somewhere between the report of transition speed reached and the point of impact the left side ESC started to emit smoke.
During its decent there was a rapid oscillation of motor RPM as evidenced by the amount of current being put to each ESC.
The FOD found after the crash contributed to an oscillating right side prop RPM and further contributed to a lack of forward momentum.
It is unclear as to why the left side ESC failed in flight but the answer lies somewhere between the full engine run up, under rated ESCs, excessive outside ambient temperature of 103degF and a failing right side motor.
The autopilot did a great job of trying to keep the bird in the air but at that critical moment when the aircraft is trying to achieve fixed wing flight the left and right side motors failed to provide enough thrust to keep it from stalling.
As the nose was pitched down from the stall, the PIC gave the aircraft full up elevator which further slowed the aircraft and thereby exacerbating the stall.
On impact a puff of smoke was seen by one or more witnesses. This emanated from the left side ESC. It is unclear exactly when that fire started or why it failed in the first place.

SO:

FOD in the right motor
Mud in the right motor
Failing left side ESC
Under rated ESCs
A transition speed of 12m/s while adequate is too close to stall speed
PID tuning may have been a contributing factor

The damaged aircraft destroyed $1000 in payloads, the aircraft itself was a totaled. Salvaged was servos, left side motor, some carbon fiber parts, power distribution board, current/voltage sensor, the autopilot, some wiring and some hardware.

I’m not reusing the motor or the autopilot.

So all in the damage was about $2000.

I’m already building gen 2.

Kywon.M_Vlogs · September 17, 2020, 8:41pm

Now that i see you used an underated ESC, i don’t even know what to say anymore.

Chad_Frazer · September 17, 2020, 9:05pm

Howdy Kwon, they are under rated only when compared to what the motors can pull. Under normal flight conditions they are about twice the capacity that would normally be seen even with strong winds.

I call them under rated due to not being able to allow a full throttle check (which is rarely needed or done) and the lower the amp capacity of the Esc the less heat it can dissipate.

The trade off is that if you have an Esc that can handle 70amps and you only ever use 5 amps your efficient use of power goes down as the circuit layout of the Esc has a range of power going in where it is has the best use of the power.

Damned if you do, damned if you don’t kind of thing

Kywon.M_Vlogs · September 22, 2020, 4:42am

Foxtech… Oof…

You know, Foxtech is only good for one thing. Making plane and VTOL frames. Nothing else.

Foxtech does not code good. Foxtech might have been the reason your craft crashed besides the motor issue. Think about it. If a stall was to occur wouldnt the plane float down without crashing down like a rock with wings? Normally, in autonomous mode, it saves itself from… uh… pretty much everything. It even cruses thermals on it’s own (I had to look it up because i found that extreamly hard to believe) So for it to not float to the ground, that’s an issue with the autopilot itself