Freezing Ardupilot

q-wertz · June 5, 2020, 1:37pm

Hello everyone,
I use a Navio2 as Rover and have issues getting it up and running.

Already posted here but emlid cannot really help…

Maybe here someone has hints.

I first encountered that the PWM output stops randomly after a short time (as described in the Link). Now for me it appears more like the complete Ardurover is freezing (but systemd still shows the process as active)

Notes:

More or less nothing is attached (No Motors, no Receiver) but also tried with attached devices
the sensors are not calibrated, the EKF runs amok but
- I do not really have the possibility to calibrate it as the connection between MissionPlanner and Rover crashes before I can do so
- that shouldn’t be a problem in my opinion for the bare output of PWM signals from my PPM input
- it shouldn’t mess up my data connection
sometimes the MissionPlanner reports that the connection is up again (but lost short time afterwards)
- tried udp and USB 433MHz Transceiver
- when I unplug or plug something (e.g. the RC receiver, GPS Antenna) it seems so “wake up” for a short time but not for a long time (-> some hardware related stuff like Failed to initialize the sensors? )
- tried MissionPlanner on windows and APM Planner 2 on Linux
Tried to compile Rover4.0.0 and put it in the place where the precompiled Rover4.0 of emlid is placed -> same behauviour
Tried it with Arducopter -> Same behaviour
Tried other RaspberryPi
Tried 2 different SD Cards
I still can connect via ssh so it is for sure not the whole raspberry which is stuck
sudo journalctl -r does not show something related (only sometimes Under-voltage detected when unplugging and replugging stuff which seems to wake it up)
Sometimes it runs for long times (up to one, two hours) but most of time just for a few seconds
Tried different powering methods (USB, Servorail, Power Module)

-> No more ideas

Some log files:

Onboard

starting rover by hand using e.g. sudo ./ardurover -A udp:192.168.178.80:14550 >> /home/pi/test.log often just stops mid line when the program seems to freeze

log1

?Problems? with AK09916 & AK8963 (barometers), also the MPU: temp reset stuff looks fishy…

Init ArduRover V4.0.0 (0e52bafa)

Free RAM: 262144
load_all took 140us
0 0 0 AP_Logger_File: buffer size=51200
AK09916: Reset Failed
AK09916: Reset Failed
AK8963: Unable to get bus semaphore
\FE	\00\00\00\00\00
A;\FE	\00\00\00\00
A\E5E\FE3\FDBarometer 1 calibration complete\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\EE\FE3\FD<startup_ground> Ground start\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\E0\CA\FE3\FDBeginning INS calibration. Do not move vehicle\00\00\00\00\86\FE	\00\00\00\00
A\\B6Init GyroRaspberry Pi 3 Model B Plus Rev 1.3. (intern: 2)
MS5611 found on bus 1 address 0x77
MPU: temp reset IMU[0] -29504 -22528
MPU: temp reset IMU[0] -27657 -5997
MPU: temp reset IMU[0] -26922 -5226
MPU: temp reset IMU[0] -23573 -5725
MPU: temp reset IMU[0] -22655 -5721
MPU: temp reset IMU[0] -21691 -8277
MPU: temp reset IMU[0] -13775 -7990
...

log2

Same Reset problems, this time lots of hex data in the log

0 0 0 AP_Logger_File: buffer size=51200
AK09916: Reset Failed
AK09916: Reset Failed
\FE	\00\00\00\00\00
A;\FE	\00\00\00\00
A\E5E\FE3\FDBarometer 1 calibration complete\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\EE\FE3\FD<startup_ground> Ground start\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\E0\CA\FE3\FDBeginning INS calibration. Do not move vehicle\00\00\00\00\86\FE	\00\00\00\00
A\\B6Init Gyro*\FE	\00\00\00\00
Ab5***\FE	a\00\00\00\00
A\88K*\FE	\00\00\00\00
Al\D4***\FE		\00\00\00\00
A\86\AA*\FE	
\00\00\00\00
A\B8)
\FE3\FDReady to drive\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\87G\FE	\00\00\00\00\00
A^#\FE
\00\00\85}ѽ\EE\B9\CD=\9B\86\90\BF\CE7\DF<8\F8\DB;\F3m\BC\E8\BD\FEJ\00\00\00\00\00\00\00\00\E1z\BF\00\00\00\80'\00\00\BC\FE3\FDGPS 1: detected as u-blox at 115200 baud\00\00\00\00\00\00\00\00\00\00\F0\B5\FE!#\00\00aX&ywZ\00\00\00]\00\00\00\00\00\00\00\00*s$\97\FE/\DC /\80 \00/\810
\00\00\00\FF\FF\00\00\00\00\00\00\00\00\00\00\00\00\FF\B3\B3\FE}/\00\00\00\00\90]\FE\98\00\00\FF\FF\92\FE*\00\00#+\FE$i\D9F\00\DC\00\00\DC\00\00\00\00\00\00\00\00\00\00\00	P\FE*A#\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\ABe\FE##\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\8Eo\FEev\D9F\00\00\00\00\00~\00\80\00\F7\FB(\00
\00\F0\FF\FB\FF
\00\00~\FEt#\00\00\00\00\00\00\00\00\00\00\00\00\00\00:\AE	g\91\FE#\00\00\C6rD\00\00\00\00-\A5&\FEePE\00\00\00\00\00aX&yw\C4F\00D\00d\00\00\00\00\00G\FE\A8\92UT\A7\00#\00\00B\8F\FE\A3\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\F1N?a\DC>\BA\AB\FE\A5/\00s\B4\FE \F1\B4\D9F\00\00\00\00\00\E4\C6>\FA\A3\DE>3%\AA>\00\00\00\00\00\00\00\00\00\00\00\00\C1\D3\FE #\00\00\8C:ʽ\\8D\D3=m\9E\91\BF\B1;=x\B5a<!\93\82\BC\B7\8C\FE!Jo<o<\EBQ\B8=ף\BC&\00\00\893\FE"\00\00uŽ\FE\EA=\F3싿%{<\9A\A3;Z\BC\A9"\FE#Jo\83:o\83:\CC\CCL\BEף\BB)\00\00n^\FE	$\00\00\00\00\00
A\82i\FE%!\00\00X&yw\F2\FE\FF\FF\EC\FE\FF\FF\00\00\F5\FF*uƿ\FE&/\DC /\80 \00/\810\00\00\00\FF\FF\00\00\00\00\00\00\00\00\00\00\00\00\FFą\FE'}&\00\00\00\00\A2\84\FE(\98\00\00\FF\FFv3\FE)*\00\00&\D3\FE*$\ABzN\00\DC\00\00\DC\00\00\00\00\00\00\00\00\00\00\00\D1\C3\FE*+A\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\A1\CC\FE,#\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\89\BA\FE-e\B7zN\00\00\00\00\00n\00q\00\FC\00\00\00\F7\FF\FC\FF\00
\97\FE.t\00\00\00\00\00\00\00\00\00\00\00\00\00\00C\FD\E7\00\FAB\FE/\00\00\8F\C8rD\00\00\00\000\ED.\FE0\D09K\00\00\00\00\00X&yw\D8F\00D\00d\00\00\00\00\AC6\FE1鯙UT\A7\00\00\00P\D8\FE2\A3\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00m\D4?f\A0t>\B8\8F\FE3\A5&\00=A\FE 4\F1\F4zN\00\00\00\00\009/\D8>:7\F1>=z\BB>\00\00\00\00\00\00\00\00\00\00\00\00\E8\B9\FE5"\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00r\B0\FE6\00\00\CE=\BC\BD\EA>\97/\86\BF\D5q\B9<\C6@;J\C7\BCF\C5\FE7JB`\E5<B`\E5<p=\8A\BE\85\EB=+\00\00\99W\FE48\F2X&yw\BA\FF\FF\FF\00\00\00\00\00\00\00\00\00\00\00\00\00\00\80?\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00]
\FE9a\00\00\E2_\BC\BD
\8B\00>\C9y\80\BF'\F4\EE;\BD9\BB\D1WQ\BB-\B3\FE:Jף<ף<)\\BE\D6x\E9=.\00\00e\D7\FE3;\FDEKF2 IMU0 initial yaw alignment complete\00\00\00\00\00\00\00\00\00\00|\A7\FE<!\00\00X%yw\FF\FF\FFQ\FF\FF\FF\00\00\F0\FFw\F14\FE=/\DC /\80 \00/\810
\00\00\00\FF\FF\00\00\00\00\00\00\00\00\00\00\00\00\FF%x\FE>}/\00\00\00\00\E5]\FE?\98\00\00\FF\FF\00%\FE@*\00\00\9A6\FEA$V\00\DC\00\00\DC\00\00\00\00\00\00\00\00\00\00\00\BC\FE*BA\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\AA\DF\FEC#\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00Ez\FEDeV\00\00\00\00\00j\00j\00\FC\00\00\00\FF\FF\FD\FF\00s\FA\FEEt\00\00\00\00\00\00\00\00\00\00\00\00\00\00C\FD\E7\00a\C3\FEF\00\00l\C8rD\00\00\00\001\8F8\FEG\90aT\00\00\00\00\00X%yw\F6F\00D\00e\00\00\00\00a\E6\FEHQQ\A1UT\A7\00\00\00\B0i\FEI\A3\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00,\8E>e\D1.>\94\81\FEJ\A5/\00$\9C\FEK\C1\00\00\00\00\00\00\00\009f\E9;\C8\E0>=\00\00\00\00\00\00Z"\FE L\F1`V\00\00\00\00\005߹>\E1\91\CC>~X\A2>\00\00\00\00\00\00\00\00\00\00\00\00\FEM\00\00\85\AD\BA\BD\9B\D7\00>Cov\BF\E5\FDL;\BFF\A9:%\F9\BB\85\FENJ\C5 0=\C5 0=\8F\C2u\BE2,>0\00\00_\C2\FEO\B2\FAX߽F\A9\DC=\E0\E4z\BF\00\00\00\00\00\00\00\00\00\00\00\00O\FE(P\B6\FAX߽F\A9\DC=\E0\E4z\BF\AE'\8CCX%yw\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00Raspberry Pi 3 Model B Plus Rev 1.3. (intern: 2)
MS5611 found on bus 1 address 0x77
MPU: temp reset IMU[0] 8194 4191

MissionPlanner

One can see the EKF running off, Unhealty AHRS, compass etc. but for me that does not explain the connection loss…
2020-06-05 12-54-06.tlog (128.5 KB)

Ideas:

Hardware problem?
- behaviour with unplugging stuff and Voltage reports are maybe hints but
  -> the emlid tools for steering a servo works
  -> more or less all provided tools from emlid do not show any error
Rover Version problem? but
-> Copter showing same problems
-> Rover 4.0 and 4.0.0 same problems
Ardupilot
- does not freeze only when everything is attached as it should?
- freezes/crashes when EKF is running wild?
I am the problem
- missing understanding of how Ardupilot works
  -> very probable, that’s why I’m asking here for help

Conclusion:

Many hours spent, confused, deprimated, feeling stupid

q-wertz · October 16, 2020, 3:32pm

Was a faulty Navio2. Bought a second one which does not have the problems above.