first of all, I have been reading this wonderful forum for years, and I learned a lot from the community already, thanks!
The reason I created this topic, is that I run out of options, since reading log files is new to me it would be very nice, if somebody could help me understand what happened with our agricultural spraying drone.
We were flying in auto mode and suddenly the drone crashed, I have been trying to understand the log files, but the only info i was able to understand, is that the EKF had some kind of issue, so it switched to IMU2 right before the crash. Can anyone please help to understand?
here is a link to the log file:
There was an odd glitch in the Motor Outputs dropping them to 0 briefly. When it recovered from this the motors on the right side of the craft were commanded to max or close to it. The motors on the left side were dropped to stabilize the craft and down it went. It would have been interesting to look at battery current at this time but it’s not logged. Does what you witnessed make sense form this observation?
thanks for your answer, your explanation is perfectly accurate. I also wanted to check the battery current, Im not sor why it is not logged. Do you have any idea why this happened? We had an error message: EKF PRIMARY: OCCURED OR FAILED TO INITILIASE.
Also, can you tell me which log reader do yo use?
Hi Károly- From the timing in the log it looks like the EKF error was generated after it lost stabilization or when it crashed. Check out the attached error messages with respect to altitude. These graph are generated from APM Planner 2.
I have to admit I have never seen an RC Output go to zero. To minimum PWM yes, but I’m not sure what 0 means frankly. Logging didn’t stop because there is other data being recorded during this event. I have looked at other logged data that might correlate but I’m not seeing anything.
However, this is rather troubling as I thought we’d fixed the safety switch issues earlier in the 3.6 series. It was found that some safety switches were activating in-flight - particularly when water was involved.
thanks, I can’t see anything in there that indicates a mavlink commanded safety state change.
I’ve also been working to test if an IOMCU reset can cause an issue with master, and it can. We had code to try to cope with it, but testing it this evening shows it wasn’t reliable.
I’ve opened a PR here to fix it:
Hi, thanks for your answer, but dont really understand the problem, why would the IOMCU reset? And what issue can it cause with the master (also what master means in this case?)? And can you explain to me the PR you opened? We are quiet new to this, and Im not sure what should we do to avoid this problem next time.
Also @peterbarker, If i understand it correctly, your solution is a bit different, can you tell me anything based on the tlog? Or if water has anything to do with it?
I think it has something with water, I was looking for correlating anomalies, and it seems that right before the safety switch engaged, the VCC power was dipped (not much, but it got all times low, during the flight). Is it possible that the water/spray got to the flight controller or cabling ?
Hi, thanks for your answer, and yes, it is absolutely possible, since we were testing a new spraying system, but It should not be problem, since we are using a drone made for spraying, and also moderate rain is ok, based on the factory guarantee policy. But if it was the water, is there any software solution? Im asking, because @tridge and @peterbarker also thinks it was a software issue, if i understand correctly.
If it is from China, then “rain is ok, based on the factory guarantee policy.” is basically zero guarantee. Unless you have conformal coating on key elements, and Silicone sealing on non movable parts… I see and serviced a couple of spraying drones from china (Joyance, TTA, ect…) and all of them were crap
The IOMCU would reset either if their was a bug in the IOMCU firmware or if there was a transient power spike which caused it to reset. It is very rare (I think this is the first time an aircraft has been lost where we think it might be an IOMCU reset) so hard to really attribute causes. We certainly have fixed bugs in the IOMCU firmware since 3.6.10.
By ‘master’ I mean the latest development branch of ArduPilot, which will become the 4.1.x release in the future. That is called ‘master’ in the source code control system we use.
The PR I opened fixes the FMU firmware to automatically disable the safety under the following conditions:
it detects an IOMCU reset (as time went backwards in the status from the IOMCU)
it knows that safety was disabled in the last 20Hz update of status from the IOMCU
the vehicle is armed
If those conditions are true then it forces the safety off to give you a chance to continue flying.
Sorry for the confusing terminology. Basically I think you hit a very rare condition. I can’t be sure if the issue is a fault in your hardware or a software bug, but I do know that ArduPilot could have done better in handling the condition, which is why I opened that PR, so if this does happen to someone else in the future it will force the safety off and should keep flying.