GPS issue with Copter-3.4-rc3 on Linux [Solved, with lessons learned]

I just upgraded my software to Copter V3.4-rc3 (e2a9772c). My previous version was Copter V3.4-rc1 (dfa90d29), pulled back in April. I’m running the BBBMini version (on a Beaglebone Black), and have had no issues with GPS until now.

On my flights from today and yesterday, I cannot enter Loiter mode, and there is no GPS telemetry in my log file. And yet when I bring my copter home and connect to it through Mission Planner via Mavlink, the GPS is clearly running, and if I collect a small amount of data, the GPS data is clearly there.

I have performed the above exercise twice (yesterday and today) and it is perfectly repeatable, i.e., works at home but not in the field. If I can figure out how to post the log files, I will.

It may not be related to V3.4-rc3. I also updated my Linux Kernel, and notice now that I need to put a 30 second delay at boot-up before starting ardupilot, otherwise it doesn’t even recognize my spi devices. My next test will be to go back to rc1 and revert my Kernel, and see which combination works.

My theory is that Arducopter isn’t immediately recognizing my GPS when it starts up (maybe Linux is taking a while before the device appears), and doesn’t check again after it becomes available. If so, is there a setting I can use to get Arducopter to continue to look for the GPS?

Has anyone else encountered these issues on a Linux platform? I’ll post what I discover too as I continue to troubleshoot.

Uploaded log files to my google drive. Here is a short one taken at home, where GPS works:

https://drive.google.com/file/d/0B43bVwzZFricYW10c1dsSDQwbjQ/view?usp=sharing

and here is one taken out in the field, where there is no GPS:

https://drive.google.com/file/d/0B43bVwzZFricb1JiQ0xzbEItamM/view?usp=sharing

Confirmed. GPS works at boot-up in rc1, does not work rc3. I have also tried reverting to the previous Linux kernel (currently on 4.4, reverted to 4.1) and that makes no difference. So I have to conclude that something has changed with rc3.

Also, if I log in, kill arducopter, and restart it, GPS seems to work. So it is something happening when I autostart it at boot time. This is why it always works for me at home. I kill arducopter and restart it with the -A udp:(ip address) flag so I can communicate with it via Mission Planner, and miraculously, GPS is working.

Any troubleshooting tips on how to get GPS working again for rc3 would be appreciated! Are there any parameters I can set to force it to continue looking for GPS?

Maybe try setting the GPS_TYPE to “2” if you’re using a ublox?
Maybe add a small delay to the ardupilot startup? Of course it would be better to know what’s changed, I’m just not sure what that is (so much has changed since -rc1 actually).

Thanks Randy! Yes, I am using a Ublox NEO-M8. Will try setting the type to 2. Also, I have already put in a 30 second delay at boot-up in my rc.local startup script before calling arducopter, giving the kernel time to set up the spi devices. I never had to put in this delay before I upgraded my Linux OS and kernel, so not happy there but that is certainly not ardupilot’s fault.

I’ll continue to troubleshoot and report back if I find anything. Just thought maybe raising this issue would ring a bell.

Incidentally, I’ve been flying all weekend, with several hours of flight time, on rc3 and the flying has never been better. Really impressive software!

I think the rc1 vs rc3 difference was a red herring. I went out to actually FLY today with rc1 (rather than just test at home) to confirm that my GPS was working. And it wasn’t. So I retract my conclusion that it has something to do with the new rc3 build. It must have something to do with my updated Linux OS and kernel, which causes the GPS to fail when I’m in the field. Maybe it has something to do with whether my ethernet cable is plugged into the BBB, I don’t know! Continuing to investigate, but looks like more of a Linux issue.

Unfortunately, I made a very serious boo boo today. I thought I could simply swap out the rc3 executable for the rc1. And the flying was horribly erratic. I’m guessing it has something to do with the updated param file, which I forgot to revert back to the rc1 version. The consequence is that I had a horrible crash, first in a year, and many broken parts including a motor shaft.

It may be a while before I can repeat a test flight. Meanwhile, if any Linux experts out there has a good idea on what might be happening, please reply.

Final update. I just solved my GPS problem, and it turned out to be a really bone-headed mistake on my part. I’m posting the solution here in case anyone else out there makes the same mistake. I was debugging an entirely separate issue, and put the following lines in my rc.local file:

#/root/arducopter-quad_3_4rc1 -B /dev/ttyO5 &
/root/arducopter-quad_3_4rc3 -B /dev/tty05 &

See the difference? I didn’t, until this morning. The /dev/tty05 is incorrect, where the zero should be the letter O. This explains why GPS worked on rc1 but not rc3. When I finally became convinced that rc1 worked, I removed the _3_4rc1 extension from the executable and used the line with the 05 in the rc.local, then went out to fly. So the GPS did not work with rc1 in that case. Worse yet, I made another serious error when I went out to fly with the old executable, which is where the real "lesson learned"comes in.

I had naively assumed that my rc3 parameter file was backwards compatible with the rc1. I have no direct evidence that this was the root cause of my crash on Tuesday, but I can think of no other explanation. My other mistake was to take off directly in Alt Hold mode, rather than try it out in Stabilize first. Reviewing the log file after my crash, I see that the calculated altitude CTUN.Alt did not track the actual altitude at all, nor was it responsive to my altitude commands. The quad was difficult to control in altitude, and ultimately crashed.

So here’s what I learned:

  • 0 and O look very similar on the PuTTY terminal window, but can make a big difference in performance
  • When upgrading the software, always save the previous parameter file in case there is a need to revert
  • Never go straight into Alt Hold or other higher mode when testing a new software build. Start out with Stabilize, and quickly return to that if anything feels funny

And just for completeness, here’s what the bad flight looked like.

The first plot shows the commanded altitude (red line, CTUN.DAlt) vs what Arducopter thought the altitude was (green line, CTUN.Alt).

The second plot shows the commanded altitude (red line, CTUN.DAlt) vs the actual altitude (blue line, BARO.Alt). Not a great flight, and broke a few parts including a motor.

Larry,
Great that you got to the bottom of it. Indeed “0” and “O” look similar!
Thanks for the reports!