Autotest SITL reboot fails on Mac

I’m trying to run an autotest on my Mac, but each test fails because each test starts with a FC reboot and the autotest script isn’t reconnecting after the reboot. The arduplane process is still running, but if I run it in lldb, it terminates.

AT-0004.0: get_parameter(STAT_RESET): PARAM_VALUE {param_id : STAT_RESET, param_value : 198805520.0, param_type : 6, param_count : 1346, param_index : 65535}
AT-0004.0: Sending param_request_read for (STAT_BOOTCNT)
AT-0004.1: get_parameter(STAT_BOOTCNT): PARAM_VALUE {param_id : STAT_BOOTCNT, param_value : 1.0, param_type : 4, param_count : 1346, param_index : 65535}
AT-0004.1: Executing reboot command
AT-0004.1: Sending COMMAND_LONG to (1,1) (MAV_CMD_PREFLIGHT_REBOOT_SHUTDOWN) (p1=1.000000 p2=1.000000 p3=0.000000 p4=0.000000 p5=0.000000 p6=0.000000  p7=0.000000)
AT-0004.1: AP: EKF3 IMU0 initialised
AT-0004.1: AP: EKF3 IMU1 initialised
AT-0004.1: ACK received: COMMAND_ACK {command : 246, result : 0, progress : 0, result_param2 : 0, target_system : 0, target_component : 0} (0.000000s)
AT-0004.1: Sending param_request_read for (STAT_BOOTCNT)
EOF on TCP socket
Attempting reconnect
[Errno 61] Connection refused sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
[Errno 22] Invalid argument sleeping
>>>> FAILED STEP: test.Plane.FenceStatic at Tue Apr 19 16:45:28 2022 (Did not detect reboot)
Traceback (most recent call last):
  File "/Users/tim/ArduPilot/master/Tools/autotest/autotest.py", line 740, in run_tests
    success = run_step(step)
  File "/Users/tim/ArduPilot/master/Tools/autotest/autotest.py", line 549, in run_step
    return run_specific_test(specific_test_to_run, binary, **fly_opts)
  File "/Users/tim/ArduPilot/master/Tools/autotest/autotest.py", line 426, in run_specific_test
    return tester.run_tests([a])
  File "/Users/tim/ArduPilot/master/Tools/autotest/common.py", line 9834, in run_tests
    self.init()
  File "/Users/tim/ArduPilot/master/Tools/autotest/common.py", line 7040, in init
    self.apply_default_parameters()
  File "/Users/tim/ArduPilot/master/Tools/autotest/common.py", line 2538, in apply_default_parameters
    self.reboot_sitl()
  File "/Users/tim/ArduPilot/master/Tools/autotest/common.py", line 1897, in reboot_sitl
    self.reboot_sitl_mav(required_bootcount=required_bootcount)
  File "/Users/tim/ArduPilot/master/Tools/autotest/common.py", line 1878, in reboot_sitl_mav
    self.detect_and_handle_reboot(old_bootcount, required_bootcount=required_bootcount)
  File "/Users/tim/ArduPilot/master/Tools/autotest/common.py", line 1913, in detect_and_handle_reboot
    raise AutoTestTimeoutException("Did not detect reboot")
common.AutoTestTimeoutException: Did not detect reboot
FAILED 1 tests: ['test.Plane.FenceStatic']

While this is happening ps -ax shows the arduplane process continues to run in the background. (i.e it’s not crashing!).

I tried the --debug flag and I got a /tmp/arduplane.log file that contains this:

Setting SIM_SPEEDUP=1.000000
Home: 37.090662 -3.074557 alt=2736.000000m hdg=0.000000
Starting sketch 'ArduPlane'
Starting SITL input
Using Irlock at port : 9005
bind port 5760 for 0
Serial port 0 on TCP port 5760
Waiting for connection ....
Connection on serial port 5760
Loaded defaults from tools/autotest/default_params/plane.parm
bind port 5762 for 2
Serial port 2 on TCP port 5762
bind port 5763 for 3
Serial port 3 on TCP port 5763
Smoothing reset at 0.001
validate_structures:469: Validating structures
Loaded defaults from tools/autotest/default_params/plane.parm
Setting SIM_SPEEDUP=1.000000
Home: 37.090662 -3.074557 alt=2736.000000m hdg=0.000000
Starting sketch 'ArduPlane'
Starting SITL input
Using Irlock at port : 9005
bind port 5760 for 0
Serial port 0 on TCP port 5760
Waiting for connection ....
Connection on serial port 5760
Loaded defaults from tools/autotest/default_params/plane.parm
bind port 5762 for 2
Serial port 2 on TCP port 5762
bind port 5763 for 3
Serial port 3 on TCP port 5763
Smoothing reset at 0.001
validate_structures:469: Validating structures
Loaded defaults from tools/autotest/default_params/plane.parm
~                                                                    

I tried putting --lldb --debug on the command line and I got this in /tmp/x.lldb

settings set target.process.stop-on-exec false
process launch

All the while “Attempting to reconnect”, the arduplane process is still running:

(base) 2022-04-19-16:40 /Users/tim/ardupilot/master % ps -ax | grep arduplane 
52145 ttys008    0:00.21 /Users/tim/ArduPilot/master/build/sitl/bin/arduplane -S --home -35.362938,149.165085,585,354 --model plane-elevrev --speedup 8 --defaults /Users/tim/ArduPilot/master/Tools/autotest/default_params/plane-jsbsim.parm --uartC=tcp:2 --disable-fgview

Here is a screen recording of a session that fails (with --lldb window).

So I’ve found that part of my problem is caused by the debugging. Specifically adding “–lldb” will cause the signal 5 (SIGTRAP) on execv(), which is what HAL_SITL uses to do a reboot.

This is actually “working” because /tmp/x.lldb options sets

settings set target.process.stop-on-exec false

Which means that even though the process lldb is debugging has stopped, the new version of arduplane has now been execed and is running with a different PID. It’s also not attached to the lldb which kind of defeats the purpose of running lldb in a window, so that isn’t going to help if reboot is being used in the test (which it pretty much is in all autotests because some params need to be set that require reboot).

Just out of interest- the SITL binary appears to be an intel format file, even though I’m compiling this on my M1 Mac Pro using the native compiler.

file sitl/bin/arduplane 
sitl/bin/arduplane: Mach-O 64-bit executable x86_64