Simulating 2 drones with SITL/Airsim in Windows/Cygwin won't work

rajat2004 · November 15, 2019, 6:15pm

Hi,
I’m assuming that since you’re trying mutli-vehicle simulation, then it’s all working properly with a single drone.
Yes, the script was written for Linux, but I think it should work for Cygwin as well, but can’t say for sure since I haven’t tested it on that. Unfortunately, I won’t be able to do so for some days atleast since recently did a fresh installation and will need to install everything again and also have exams right now.

Yes, this IP address (239.255.145.50:14550) seems very strange, on digging around a bit found this but still have no idea why

github.com

ArduPilot/ardupilot/blob/49069c7ad56c2a6280fc7db62134e48a382c37a7/Tools/autotest/sim_vehicle.py#L928


                     default=False,
                     help="Don't launch MAVProxy")
group_sim.add_option("", "--fresh-params",
                     action='store_true',
                     dest='fresh_params',
                     default=False,
                     help="Generate and use local parameter help XML")
group_sim.add_option("", "--mcast",
                     action="store_true",
                     default=False,
                     help="Use multicasting at default 239.255.145.50:14550")
group_sim.add_option("", "--osd",
                     action='store_true',
                     dest='OSD',
                     default=False,
                     help="Enable SITL OSD")
group_sim.add_option("", "--tonealarm",
                     action='store_true',
                     dest='tonealarm',
                     default=False,
                     help="Enable SITL ToneAlarm")

Anyways, you could try specifying the IP directly, maybe something like mcast:127.0.0.1:14500, hopefully the format is correct but might need some modification

For MAVProxy, please have a look at this - https://ardupilot.github.io/MAVProxy/html/getting_started/download_and_installation.html#windows

Hope this helps!

josepeck · November 19, 2019, 12:00pm

Hi thank you for the quick answer,

I specified the mcast:127.0.0.1:14550 but still getting the multicast error. However, the first time i ran it, i got the two drones in Airsim visualized (image below) but Unreal was stuck. I’m getting a continuous output “No sensor message received - Bad file descriptor” (i looked in sim_vehicle.py and couldn’t find this error). Do you really think that the Windows User folder whitespace could be the problem? @rajat2004

This is my console output right now (not the one with the two stuck drones, that one was similar but after a moment, the “No sensor data” error stopped and the two drones popped up)

Starting SITL Airsim
Bind SITL sensor input at 127.0.0.1:9003
AirSim control interface set to 127.0.0.1:9002
Starting sketch ‘ArduCopter’
Starting SITL input
Using Irlock at port : 9005
UDP connection 127.0.0.1:14550
Loaded defaults from /home/copter.parm,/home/quadX.parm
Home: -35.363262 149.165237 alt=584.000000m hdg=353.000000
Starting copter 1
No sensor message received - Invalid argument
~/ardupilot/copter1 ~/ardupilot
~/ardupilot
Starting SITL Airsim
Bind SITL sensor input at 127.0.0.1:9013
AirSim control interface set to 127.0.0.1:9012
Starting sketch ‘ArduCopter’
Starting SITL input
Using Irlock at port : 9015
bind port 5897 for 0
Serial port 0 on TCP port 5770
Loaded defaults from /home/copter.parm,/home/quadX.parm,follow.parm
Home: -35.363262 149.165237 alt=584.000000m hdg=353.000000
UDP multicast connection 127.0.0.1:14550
multicast membership add failed on port 14550 - Cannot assign requested address
No sensor message received - Bad file descriptor
No sensor message received - Bad file descriptor
No sensor message received - Bad file descriptor
No sensor message received - Bad file descriptor
No sensor message received - Bad file descriptor

rajat2004 · November 19, 2019, 2:38pm

@josepeck Well, certainly the whitespace isn’t the issue, or the binaries themselves wouldn’t have been located and executed
The “No sensor message received” message is being generated from the AirSim backend in ArduPilot

github.com

ArduPilot/ardupilot/blob/master/libraries/SITL/SIM_AirSim.cpp#L223




/*
	Receive new sensor data from simulator
	This is a blocking function
*/
void AirSim::recv_fdm()
{
    // Receive sensor packet
    ssize_t ret = sock.recv(&sensor_buffer[sensor_buffer_len], sizeof(sensor_buffer)-sensor_buffer_len, 100);
    while (ret <= 0) {
        printf("No sensor message received - %s\n", strerror(errno));
        ret = sock.recv(&sensor_buffer[sensor_buffer_len], sizeof(sensor_buffer)-sensor_buffer_len, 100);
    }


    // convert '\n' into nul
    while (uint8_t *p = (uint8_t *)memchr(&sensor_buffer[sensor_buffer_len], '\n', ret)) {
        *p = 0;
    }
    sensor_buffer_len += ret;


    const uint8_t *p2 = (const uint8_t *)memrchr(sensor_buffer, 0, sensor_buffer_len);

The Bad File descriptor is due to the socket not being initialized properly, and the Unreal Engine is stuck since it’s waiting for messages from Ardupilot. I’ll try to test it as soon as possible on Windows, if possible, could you try using WSL once and see if the problem is occurring there also, if yes then atleast we’ll know it’s mostly related to Cygwin.

From the new output you’ve posted above, it’s trying to bind the multicast socket to 127.0.0.1:14550, so the command-line value does work, try and see if changing the port makes any difference

All of my testing on Windows was with Ardupilot running inside WSL, I don’t exactly remember if I had tested multi-vehicle simulation also but hopefully I did
Anyways, I’ll try to check this and will post if there’s any progress

josepeck · November 20, 2019, 10:45am

@rajat2004 I tried with WSL and this time i got a Windows Firewall prompt (maybe there is the issue). I gave permissions to ardupilot and then got the same error (but this tim, the original follow-copter.sh didn’t gave me the multicast error).

rajat2004 · November 20, 2019, 2:51pm

@josepeck I’ve got it working finally! Got the same problems as yours, figuring it out was a long and screwed up journey from WSL issues to networking pages, but it’s working now!!

I’ve pushed my changes to this branch - https://github.com/rajat2004/ardupilot/tree/airsim-multi-windows-fixes
Cleaning up and opening a PR will take some time, but hopefully soon.

There were a lot of problems on the net with WSL networking and such, as to what is supported and all, but then from the commits above you can see how small the changes actually were, it’s funny and sad as well.
The main thing whch put me on the right track - https://github.com/rust-lang/rust/pull/21267

So it seems like multicast on Windows works on 127.0.0.1, others are blocked by firewall. Though my testing was with the Firewall disabled, so might need to do that.

Please test the changes on your system and see if it works, in WSL as well as Cygwin. I didn’t use Cygwin, but maybe might work.

Thanks for reporting the issue and hopefully works on your system as well!

ppoirier · November 20, 2019, 3:40pm

Interesting, I just started to experiment on AirSim … and I like it … Thanks

I loaded WSL and experience the same issue on address:
Starting SITL Airsim
Bind SITL sensor input at 127.0.0.1:9013
AirSim control interface set to 127.0.0.1:9012
Starting sketch ‘ArduCopter’
Starting SITL input
Using Irlock at port : 9015
bind port 5770 for 0
Serial port 0 on TCP port 5770
Loaded defaults from /home/alien/ardupilot/Tools/autotest/default_params/copter.parm,/home/alien/ardupilot/libraries/SITL/examples/Airsim/quadX.parm,follow.parm
Home: -35.363262 149.165237 alt=584.000000m hdg=353.000000
UDP multicast connection 239.255.145.50:14550
UDP multicast connection 239.255.145.50:14550
multicast bind failed on port 14550 - Cannot assign requested address
multicast bind failed on port 14550 - Cannot assign requested address

Where do you change this adress to 127.0.0.1 ? connection 239.255.145.50:14550

rajat2004 · November 20, 2019, 3:47pm

@ppoirier I changed it in the follow_copter.sh script - https://github.com/ArduPilot/ardupilot/commit/9e37e92b38e6f6debf07cbcc4afbff97f512a22f

Glad to know that you like it, I was thinking of starting off with some Vision & Lidar based experiments with it after my exams end, any ideas or suggestions would be great

ppoirier · November 21, 2019, 12:28am

OK I found a method with TCP on WSL2

A) You start the copter the usual way:
libraries/SITL/examples/Airsim/follow-copter.sh 127.0.0.1

B) and you start mavproxy this way:
mavproxy.py --master=tcp:127.0.0.1:5763 --master=tcp:127.0.0.1:5770 --source-system=1 --console --map

and we have 2 friends flying

rajat2004 · November 21, 2019, 4:08am

@ppoirier Great! Did you have to change some things in the follow_copter.sh script?
Since you’re using TCP, I think you would have modified this line to remove the udpclient and add tcp here (Also, now I can see that the udpclient problem would have been happening here, the $GCS_IP might have been empty)

github.com

ArduPilot/ardupilot/blob/master/libraries/SITL/examples/Airsim/follow-copter.sh#L17


GCS_IP=$1


BASE_DEFAULTS="$ROOTDIR/Tools/autotest/default_params/copter.parm,$ROOTDIR/libraries/SITL/examples/Airsim/quadX.parm"


[ -x "$COPTER" ] || {
	./waf configure --board sitl
	./waf copter
}


# start up main rover in the current directory
$COPTER --model airsim-copter --uartA udpclient:$GCS_IP --uartC mcast: --defaults $BASE_DEFAULTS &


# now start another copter to follow the first, using
# a separate directory to keep the eeprom.bin and logs separate
# for increasing the number of copters, change the number in seq
for i in $(seq 1); do
    echo "Starting copter $i"
    mkdir -p copter$i


    SYSID=$(expr $i + 1)
    FOLL_SYSID=$(expr $SYSID - 1)

Also, is the mcast enabled? I’m not sure since both the vehicles are in Guided mode rather than the second one being in Follow mode where it’ll be needed

josepeck · November 25, 2019, 12:29pm

Thank you @rajat2004!!, I got the two copters now flying finally (through connecting them one by one in MissionPlanner). Now i’m trying to attach mavproxy as the doc webpage suggests but I get an ‘X display’ error and the console nor the map are showing. Do you know what this might be @ppoirier?

I actually found mavproxy.py in the.local directory but not in /ardupilot as the documentation suggests.

ppoirier · November 25, 2019, 12:42pm

Hello,

You need a Graphic Server for Linux.
I installed Xming X Server for Windows and it works pretty fine.

josepeck · November 26, 2019, 9:53am

Hi @rajat2004, I tried the fixes on Cygwin (without firewall) and still getting the error and the two freezed drones. Its working fine in WSL.

rajat2004 · November 26, 2019, 10:07am

@josepeck Ahh, bad luck, maybe it was too much to hope that it would work on Cygwin as well.
Could you do a short test and see if binding on 0.0.0.0 works? (mcast:0.0.0.0:14500), both with and without the fix, and see if that makes any difference?
Just a thought which I had, I don’t have Cygwin setup right now, will take some time to get that up and running.

josepeck · November 26, 2019, 12:31pm

Thank you! I got it now. Still I haven’t been able to get the second drone to follow the first one.Did you managed? I don’t get a connection with mavproxy when running mavproxy.py --master=tcp:127.0.0.1:14550 --source-system 1 --console --map like in the docuementation.

I get

Connect tcp:127.0.0.1:14550 source_system=1
[Errno 111] Connection refused sleeping
[Errno 111] Connection refused sleeping
[Errno 111] Connection refused sleeping
[Errno 111] Connection refused sleeping
Failed to connect to tcp:127.0.0.1:14550 : [Errno 111] Connection refused

ppoirier · November 26, 2019, 12:34pm

I used these ports - that are the standard tcp ports for SITL.
But I cannot get the multicast UDP started , I suspect there is some kind of problem wit this multicast script… but I did not looked at it yet

rajat2004 · November 27, 2019, 3:39am

Maybe try --master=tcp:0.0.0.0:14550 since that worked for multicast in Cygwin?

rajat2004 · November 27, 2019, 3:42am

Based on this discussion, there can be quite a lot of improvements to the multi-vehicle script, atleast so that things don’t just stop working

I’ve opened a PR -

Would be great if you could test the PR! Please do comment for more changes, fixes, etc

josepeck · November 27, 2019, 12:51pm

@rajat2004 I tried your new script and it’s working fine in WSL even with 3 drones. But when i try to run mavproxy with the port 14550 i get this error. It seems that the only ports that i can get a heartbeat from the two drones are 5763 and 5770 like @ppoirier did.

jose@GN-PC-22-W:~$ ./.local/bin/mavproxy.py --master=0.0.0.0:14550 --source-system 1 --consol
e --map
Connect 0.0.0.0:14550 source_system=1
Loaded module console
Loaded module map
Log Directory:
Telemetry log: mav.tlog
Waiting for heartbeat from 0.0.0.0:14550
** MAV>**

rajat2004 · November 27, 2019, 2:39pm

@josepeck Thanks for testing! The MAVProxy problem is with WSL and Cygwin both or with Cygwin only?

Also, from the picture, it seems that the Use Less CPU option is enabled in Unreal Engine.
Not sure if that’s intentionally, but if not then do the following: Go to Edit->Editor Preferences, in the Search box type CPU and ensure that the Use Less CPU when in Background is unchecked.

This will keep the graphics and everything same when switching between windows.

josepeck · November 28, 2019, 8:33am

@rajat2004 I switched to WSL, testing in Cygwin for me is a pain because of the whitespace in my Windows Users folder name (no easy task to remove). In order to test in Cygwin i need to modify follow-copter every time and change the ROOTDIR variable.

I reduced the graphics because the computer i’m working on is really slow and so far I don’t need the visuals.