I built my first arducopter quad with 9’’ props, firmware 4.0.3, FC Pixracer, GPS+MAG BN-880, FPV and control openHD.
It often in RTL mode only loiters and is not returning. It seems to work better closer to home and lower, mostly not working far and high.
Sometimes I’m getting strange GPS glitches even I can’t see any reason for them - I reworked several times copter construction to avoid any possible interference to GPS or MAG and even replaced GPS+MAG for another type (GR-BD), re-calibrated compass at flying field, but still the same. GPS had all the time between 16-19 locked satellites and track in .KML is very consistent without any strange jumps.
By side the GPS glitches which I don’t believe they’re caused by GPS wasn’t RTL working even in period when no GPS glitch was present (e.g. 17:45:57-17:49:41 in my log.).
OSD shows RTL mode, but copter loiters and not returning (e.g. 17:46:29) - see zero speed.
At last time I engaded RTL at 17:51:43 and copter was loitering until 17:52:06 (so 23 seconds) and then did proper RTL with landing at home position. Why it waited so long?
My WPNAV_SPEED is set to 500.
My quad construction: https://drive.google.com/file/d/1riphAWwbpuzMSDtkM3S7iylepeuFfYrY/view?usp=sharing
BIN logfile: https://drive.google.com/file/d/1Qller-3MC_BfLa3fVu0qacM9K2DCi74h/view?usp=sharing
logfile screenshot overview: https://drive.google.com/file/d/1ITYRHM7A-K_3wVpF10OT3wndygF85Az_/view?usp=sharing
Please, can you give me some advice? Really don’t know what to do now.
Thanks!
Just quickly:
vibrations are high and there’s clipping
Vcc is low - ensure there isnt a telemetry radio powered by the serial port Vcc, or get a new power brick
I’ll check more soon
EDIT: the GPS glitches are occurring around the same times as high vibrations - which means arducopter is losing track of it’s position (due to the vibes) and cant tell if the GPS info is correct.
More edits related more directly to RTL:
You’ve got TERRAIN_ENABLE,1 and WPNAV_RFND_USE,1 but there’s 0 terrain tiles loaded and hundreds pending, and no evidence of a rangefinder. Probably set these to 0.
After you fix the vibrations (prop balancing, wiring vibrating the FC, FC mounting) then run through this tuning guide
https://ardupilot.org/copter/docs/tuning-process-instructions.html
And use this helper spreadsheet, also pay close attention to the “Suggested” settings - probably should be mandatory 
Looking at the pic, you might need to stiffen up those centre plates - the extended holding GPS and RC radio are probably vibrating like crazy - even if you cant see it.
Thank you for many interesting ideas - I’ll check all the recommendations.
Just to clarify few points: I used for learning arducopter older quad frame I had at home built in C&D concept, this means top board with battery, FC and GPS is mounted on rubber dampeners and “vibration insulated” from the bottom frame with motors. Vibrations on the electronic board aren’t therefore that bad, or at least I think so. (Reading my log, do you think vibrations at FC are abnormally bad?)
At least flying attitude of the quad is subjectively quite good with default PID but I can’t effectively use PID autotuning because it always ends up with nonsense unflyable results (maybe effect of dampened frame?)
In previous version was RC receiver and GPS swapped but glitches were about the same.
Black board with white antenna at highest point is in fact WIFi adapter (openHD uses WIFi adapter for all - telemetry, RC and video downlink.)
GPS with compass I moved several times and now is at the most remote part available on a small (black) EPP block. Both these shouldn’t mind some reasonable vibrations, do they?
Small circuit on the front dampened table with red antenna is tBeacon lost model finder and inflight is completely passive so it shouldn’t make any difference.
I’ll try to disable TERRAIN_ENABLE,1 and WPNAV_RFND_USE,1, I just thought if they aren’t found, they are just omitted.
Once more thank you for all ideas, I’ll keep posting my findings.
Roman
Today I made another flight with more success 
I replaced Pixracer FC for another piece of exactly the same board, copied settings (only recalibrated ACC), disabled TERRAIN_ENABLE and WPNAV_RFND_USE as xfacta advised (thanks!) and connected notebook with Mission Planner and with a friend at the display.
During test flight I engaged RTH four times at different distances and it always worked! I don’t know whether the difference was in another FC or in disabled terrain and RFND, but all worked now as it should.
I got only one GPS glitch while I was flying with maximal speed (about 50 km/h, which is quite fast for 9x4.5 props), at this speed also vibrations extremely increased so I think GPS glitches are coming mainly from vibrations.
These vibrations are probably aerodynamic so it can’t be easily removed (maybe different motors, props…) but this is just a test machine so I’m not going to make a Ferrari of it 
.
For me it is important to know reasons so for now I’d accept it as a explanation and will play with the quad as is with limitations I know. Thanks for all the support, any further ideas wellcome, I’ll check this topic time from time…
Roman