I am flying a plane that has been flown in the past without issues, is running an original 3DR Pixhawk and GPS, and this morning the first flight of the day was perfect. Firmware is Arduplane 3.5.3 and logs are below.
First flight was a basic take off, autocal for the airspeed sensor as the air in this location is much denser than where we initially calibrated it, already redid the compass calibration for this aircraft in the new location as well. First flight was a take off to 200 meters, RTL for 8 min, then a basic auto mission and auto landing and it flew absolutely perfect.
Landed, setup the camera on board, recalibrated the airspeed sensor, and took off again without cycling the power. The take off was fine and reached 200 meters as expected, then the plane turned toward the first waypoint and all was good. Once it got close to the first waypoint is when it started acting very weird, the plane would pitch up very sharply and try to turn. After seeing this I put the plane into RTL to get it back to land, however, it started doing a very tight loiter circle in a totally different location. From here I tried guided mode to get it back in comfortable sight, however again, it tried to pitch up and ended up stalling itself. I switched to manual, recovered the stall (simply giving throttle and pitching up) then switched to FBWA to get it back safely and landed. Only at one point in FBWA did the plane try and pitch up again, however, I was able to get it back and land safely.
I am very concerned as I have never seen this behavior before in Arduplane and have built quite a few planes without issues. Any help or suggestions would be greatly appreciated. I reviewed the log and there are no signs of mechanical issues as the desired pitch and roll seem to always match up.
I think I had a similar behavior lately, see my latest post “Bixler 2 - Crash after…”, about a week ago.
Comparing your log image from the final part of your flight, the deviation between your ATT.DesRoll/ATT.Roll curves looks similar to my ATT.DesPitch/ATT.Pitch curves. I enclose an image showing this - your struggle to successfully save the plane (good job!).
But there is a big differnce.
Your log also shows some peculiar behaviour earlier in the flight during AUTO, RTL and FBWA - matching your descripton. There are some violent, commanded jumps on both ATT.DesRoll and ATT.DesPitch. I guess these must be coming from the controller, at least in AUTO mode. See enclosed image.
Sorry, I have no good explanation for this, but I hope some more knowlegable could have a look at our logs and perhaps find an explanation. Of course, there may be different issues that I have not thought of.
I have figured out the problem, however, I am very concerned as there is no warning signs for the issue.
The issue was a 1.3 ghz 1500mw VTX I was using causing interference with something. I don’t know exactly what, but, I did 5 more flights yesterday with and without the VTX. Every single time when the VTX was plugged in, even though it is in the nose of the plane and I also tested on the far wing, it causes the autopilot to go aboslutely crazy in Auto mode after it switches out of take off and tried to head to the first waypoint. Also, even in FBWA the auto pilot can go crazy so it is somehow effecting the attitude control. I was able to recover the plane every time except for one, however, by some miracle, the plane had not a single scratch on it when it crash landed about 600 meters from the starting point.
The issue I have is there is no warning or way of detecting that this is a problem before take off. All sensors look good, all controls function properly, and it passes all preflight checks with flying colors.
If you look at that same log, there is 2 flights, the first flight is without the VTX and flies perfect. The second flight is with the VTX and has all the issues. Is there anything in the log between the two flights that would indicate interference that would cause this erratic behavior?
I am glad we found the problem, however, it is scary that such a fatal interference can go undetected. Hopefully we can find an indicated to add as a preflight check for interference.
You are very welcome!
I am glad you found the cause, and I agree about the lack of warning. This is complicated stuff, so it may be the nature of it, noone can foresee every issue in all those situations…
More importantly the GPS’s reported velocity accuracy numbers. With this level of noise on the GPS it’s not surprising there was a problem. I had thought this was validated in the prearm checks hoever. But I since there is no data logged between the two flights its hard to validate exactly what the arming check was seeing.
There is definetly GPS interference, however, if it is bad enough to cause a fatal flight, why was it allowed to arm?
With the degraded sat count we still had 9-11 sats which is enough for safe flight.
My biggest concern, to be honest, is the issues in FBWA. The plane pitched up and stalled itself in FBWA without my command to do so which leads me to believe the GPS is not the only thing effected.
The transmitter was placed in the nose of the plane and the pixhawk is near the tail so there is quite a bit of seperation from the two.
I haven’t looked at the rest of the log, but what that graph is showing you is that the satellite count is not nearly sufficient to be trusted as the indication that you are good to fly. The accuracy numbers being reported from the autopilot were quite far from being good.
I haven’t had a chance to look at the rest of the log yet, but I think it’s worth emphasizing that a pure satellite count isn’t a indicator of the stability of the solution, nor are the DOP values.
@Jman841 When did you turn on the VTX? The EKF should have bounced the arming requests do to the horizontal accuracy problems. Can you reproduce this with logging while disarmed?
I’ll give it a try tomorrow to reproduce, however, I have logs with no VTX and logs with the VTX. I’ll download them off the plane tomorrow and post them for you to compare. This plane is for a customer so I am going to swap out the VTX with a different one tomorrow and test then I won’t have it anymore for further testing. But, hopefully these logs will be sufficient.
I honestly do not remember exactly when the VTX was turned on, however, the flight with no VTX/issues as labeled, is an entire flight without the VTX and the other two definitely have the VTX powered on during flight, so from the point of arming until the point of landing the VTX is definitely powered on to use for comparison.
Part if what I’m trying to chase down here is that on the original problem log I can’t see why the arming checks let you arm. (Unless you had a momentarily low accuracy) unfortunately without log while disarmed this is impossible to tell. I’ll try and look through the other logs today.
Depends on the version. If you are flying master (and maybe 3.6.0 beta?) there is a new parameter LOG_DISARMED (set to 1 to log while disarmed). There is also a new LOG_REPLAY (set to 1 to log the information needed to do replays of the sensor data for looking into EKF bugs). Otherwise its still in the bitmask same as it has been.
EDIT: If memory serves thats set it to 131071, but I could be wrong, (and am not near a aircraft to test at the moment).
Here’s the log from today, this is with the VTX powered on the entire time from connecting the battery until the crash. This is after moving the GPS to the tail of the plane, about a meter away from the VTX. I am skeptical that the issues is GPS interference at this point. Possibly the VTX is interfering with other sensors? It is close to the airspeed sensor so maybe putting interference into the I2C port?
The only difference with this plane and all the others running the same VTX is the pixhawk is a little bit closer to the VTX (About 10 cm closer) and this is running Arduplane 3.5.3 with the others running 3.5.2.
Was there any other change in 3.5.3 that would cause more issues with EKF errors?
What other sensors could cause these issues that we can detect in the logs?
1.3Ghz VTX is well known for interfering with GPS, unless you can fly exclusively on GLONASS, try to get more distance between the devices, or a less noisy, or lower power VTX.
