We have built several flying wings for a humanitarian project in Africa.
Historically they have been working fine but yesterday two of them crashed consecutively due to what looks like a bug/software issue,
I’ve attached logs from the last crash. In the data you can clearly see several ‘hiccups’ in the pitch and altitude data.
The GPS signal is very good and continuous.
The airspeed data also looks good.
The Barometer seems to agree very well with GPS.
Yet the POS.RelHomeAlt has severe jumps back and forth, seemingly causing instabilities in pitch and roll control inducing crashes.
This issue exists in both full Auto and also FBWA.
We are running Arduplane 3.9.2, on a Pixhawk 2.4.8.
We have an airspeed sensor (Pitot tube), and external GPS/compass antenna/sensor.
Please advice on what could be causing this.
We have noted that the NKF5.offset seem to jump along with the errors in altitude estimation.
RIght now we cannot fly any more missions until this issue is solved, and we only have access to the drone corridor here in Malawi for two more days.
If your sensors are clipping the EKF will get unhappy. Suggest checking previous flights to see if there is a correlation between throttle output and innovations. My current best guess is that your prop is off-balance or motor on the way out.
Wow, thanks for the fast reply!
I forgot to mention that during the manual phase the aircraft was still on the ground. I guess it is obvious for you but just to leave nothing uncertain.
When you say our sensors are clipping, you are referring to the onboard IMU, gyros and acc?
Vibrations would cause the acc to bottom out in its range no longer supplying reliable data, right?
The fact that DCM and EKF do not agree, and that the ATT.pitch (Assuming EKF?) seems more reliable means that the error is dependent on the ACC data, since the DCM does not use it?
This is my first real digging into arduplane and ardupilot, have only ever built this stuff myself previously.
RIght now we are looking into vibrations from the prop, and it does seem like it could be an issue.
Switching into manual mode and turning the prop off should allow us to glide to a land, right?
Many many thanks for all your help!
Update:
We went through some older logs of successful flight to look for glitches, we also put the crashed aircraft on a stand, spun up the motor and checked for vibrations.
The sucessful flight do show the same glitches happening, albeit smaller in magnitude.
The clipping was not a problem in those.
The vibrations when running the motor with the aircraft on a stand did not cause clipping, even when we switched the prop to a clearly unbalanced one (at least compared to the one we were running at the crash).
Wow, thanks for the fast reply!
I forgot to mention that during the manual phase the aircraft was still on the ground. I guess it is obvious for you but just to leave nothing uncertain.
Yep, I usually plot some altitude to get an idea of when the vehicle is
flying.
When you say our sensors are clipping, you are referring to the onboard IMU, gyros and acc?
Accelerometers.
Vibrations would cause the acc to bottom out in its range no longer supplying reliable data, right?
Correct.
The fact that DCM and EKF do not agree, and that the ATT.pitch (Assuming EKF?) seems more reliable means that the error is dependent on the ACC data,
since the DCM does not use it?
Yes.
RIght now we are looking into vibrations from the prop, and it does seem like it could be an issue.
Switching into manual mode and turning the prop off should allow us to glide to a land, right?
Yes indeed. This is one of the reasons being able to pilot a vehicle
manually is handy
Wow, thank you so much! Then the testing we did was the right testing. It does leave us wondering though why the glitches happen on aircraft that do not saturate/clip the acc. The glitches are benign enough to not cause crashes on those aircraft, and are then maybe just part of the system.
But it leaves me.wondering if there is no problem anymore now that we balanced the prop, or if the underlying issue may still be present.
Did you see the update I added to my last post? What did you think of that data?
If you find nothing strange there we will sort through the debree and make sure the prop on the aircraft was indeed very badly balanced (because on a stand an unbalanced prop was unable to.reproduce the clipping, but maybe effects are much worse in free air than on a stand; the stand may act as a low pass filter)
If so then we can still make our big target today and things will be awesome.
Otherwise we will try to build a sacrificial aircraft, send it up and see if we can reproduce the issue.
We tried the increased rate, but all logs from those flight come back with empty binaries and/or missing headers.
We had a few more crashes, with no clipping or vibrations causing the innovations to flip out.
One of the crashes were due to roll reference being 90 when switching from FBWA to auto.
The goal for this project was not reached fully and we will be looking into the data more thoroughly now that we no longer can fly at the corridor.
Coworkers in USA have had similar “plane dropping out of the sky” problems with the pixhawk and arduplane, we are sharing our logs to see what could be up. Even though the plane was in excellent condition at takeoff.
If we find something we will post it, else we might just need to build our own controller from scratch.
Many thanks for you help, have a nice day!
//Robert
Robert, you mentioned Pixhawk 2.4.8. Where did you source them? As far as I know the 2.4.8 is a cost cutted mass produced version from China, which usually built from questionable quality components. If this is the case, then I would not rule out the hw/sensor malfunction as well…
