I have recently done some test flights with my Pixhawk powered Y6. The flights were typically either vertical (going up and down at 300m away from me) or in a circle around me (also at about 300m radius). The 4th flight was one of the horizontal circle flights and I repeated it for the 5th flight (exact same flight path). This time shortly after switching to Auto the copter seemed to loose altitude and then started to roll and pitch violently after crashing into the trees about 20m below it.
After viewing the data log I could see that both GPS receivers lost all their satellites at once, then one of the receivers regained satellites for a short period after which it lost again all satellites. Shortly thereafter I assumed it must have crashed as the logs just stopped. The GPS receivers were from two different manufacturers, the first being the Ublox 3DR receiver and the second a RTK receiver running as a normal GPS receiver.
My question is obviously, what happened and how can I prevent this from happening again.
I just flew the flight plan and repeated the exact same one. If it was external influences then why did all go well the first time. How can it just instantaneously loose both GPS receivers like that. Furthermore if I lost GPS signal the parameter FS_EKA_ACTION = 1 which means it should then just have landed right. Maybe it did try to land but how can I see that from the data log? Apparently there should be an ERR message showing the fail-safe condition but I can’t find it.
I am attaching the log if anybody has a minute to look at it. I have spent a few hours with it but can’t get much more from it other than that the GPS receivers stopped and then the accelerometers shows large deviations from the desired values.
A quick look indicates GPS problems that also seem to end up influencing EKF results, which is probably what caused the crash.
This concerns me as I am fitting an RTK GPS to an Octo at the moment.
What make of RTK GP{S are you using?
From the log:
Altitude Estimate Divergence between POS.Alt and GPS and GPS2 is happening a lot.
Is the Pixhawk bar exposed or subject to pressure changes?
The attitude controller is also having problems with “The autopilot reports both the craft’s attitude and the attitude the craft believes it should be at” with a lot of motor clipping both low and high.
At first it is short duration with no motor clipping but once the duration gets above 3seconds the log indicates motor clipping.
How well tuned is this copter?
Could it be getting into oscillations that increase to the point of non recovery?
Motors 2-4-6 seem to be clipping high the most towards the end.
The log analyser is also indicating “Severe scheduler overruns” so the flight controller seems to have really lost its shit before crashing.
I have seen in the forums where the Dev’s recommend not to use 2 GPS’s at the moment.
I hope this is corrected soon but at present I will be setting the RTK GPS as the only external.
By the look of it the internal schedular can’t handle it and flips out, which may be the cause, or just a contributing factor combined with the tune.
To get a better idea of what went wrong and also allowing me to read future logs, could you help me in understanding the issues you mentioned.
“Altitude Estimate Divergence between POS.Alt and GPS and GPS2 is happening a lot”:
I plotted POS.Alt, GPS.Alt and GPS2.Alt and the two GPS’s seems to follow each other fairly close while POS seems to have an offset. I thought this would be normal since POS is a filtered value and would differ from the GPS values. In some cases I noted a small difference between the GPS and GPS2 altitude but I suppose this could be due to the receive sensitivities of the GPS receivers that might be different (one also receiving GLONASS and GPS while the other just receive GPS). Could you elaborate more what you meant with this statement? The Pixhawk was enclosed inside the frame of the copter so there would normally not have been large pressure changes, it was also not really windy or cloudy on the day.
The attitude controller is also having problems with “The autopilot reports both the craft’s attitude and the attitude the craft believes it should be at”:
Where can I see this, I looked at ATT.DesRoll compared with ATT.Roll and did the same for pitch and yaw. Here I can also see that at the end the Pixhawk could for sure not control the copter anymore.
“lot of motor clipping both low and high”:
I checked RCOU.Ch1 through to RCOU.Ch6 and can see that all channels clip at the bottom during landing and like you mentioned 2,4 and 6 clips towards the end. Channels 2,4 and 6 are from the top mounted motors or the Y6B frame. Is this clipping a bad thing or does it just show that the motors worked at full throttle?
I tuned another copter which is supposed to be exactly the same and just copied the parameters to this one, maybe not the wises thing to have done I suppose. I think you are right about the copter going into an oscillation from which it could not recover but what caused it to go into this state?
“Severe scheduler overruns”:
Not sure where I should look for this, maybe the PM logs should provide more details. I guess this means some tasks took longer than the scheduled time causing other tasks to slow down. Then like you mentioned the autopilot could probably not perform critical tasks fast enough causing the crash. It would however be interesting to know when this severe scheduler overrun occurred and why it “triggered” at that moment.
As a side question, how can I see from the logs if a fail-safe condition was triggered. Apparently there should be a ERR log but I can’t find it?
This is a very valuable exercise for me in understanding the autopilot behaviour better, thanks for the very valuable input and help that you provided! I also hope that we could soon run 2 GPS units without any surprises. For now I also think you have the correct approach in using just a GPS and making it RTK.
I am trying out a log analysis tool that you can find here: drone kit-la
It appears there are a lot of issues being shown up with your logs, so I would suggest installing drone kit-la or other tool and having a look for yourself.
When plotting the log in your GCS you should see any fail-safe conditions highlighted
The overruns seem the biggest culprit and the clipping could, could, be from being under powered.
I’m just guessing here.
I would disable one GPS and see if you get these overruns in a test flight.
That sounds great, I have just downloaded it and will check it out!
Will also direct my efforts in the direction of the overruns, this seems like a critical aspect and would be great to understand how it gets influenced in more detail.
Thanks for your help and pointing me in the right direction.