AirSim + Multiple ArduCopter SITL + MAVROS

ArduCopter Copter Simulation
1

Hello,

I am trying to fly multiple quadrotors each led by its own ArduCopter SITL instance in AirSim simulation and also aim to control them through MAVROS topics and services with a couple ROS nodes. I do not start any kind of GCS.

I examined how ArduCopter SITL runs through terminal environment and came up with the following (I slightly modified the libraries/SITL/examples/AirSim/follow-copter.sh so that each instance would be controlled by its own SITL, instead of following the quad with SYSID - 1):

for i in $(seq $NCOPTER); do
    SYSID=$(expr $i - 1)
    echo "Starting copter $SYSID"
    mkdir -p copter$SYSID
    port=$((14550 + $SYSID * 10))

    cat <<EOF > copter$SYSID/identity.parm
SYSID_THISMAV $SYSID
EOF
    pushd copter$SYSID
    $COPTER --model airsim-copter --uartA udpclient:$GCS_IP:$port --uartC mcast:$MCAST_IP:$port --instance $SYSID --defaults $BASE_DEFAULTS,identity.parm &
    popd
done
wait
  • MCAST_IP is default multicast IP
  • GCS_IP is just 127.0.0.1
  • NCOPTER is the argument for how many quads I want to fly/simulate in AirSim

I have managed to run multiple SITLs with distinct UDP ports and connections. However, I have a problem with connecting MAVROS and properly using ROS mechanism. For example,

I have 2 quads with the following distinct arguments:

  1. –uartA udpclient:127.0.0.1:14550 --instance 0
  2. –uartA udpclient:127.0.0.1:14560 --instance 1

I launch first MAVROS with the following:

roslaunch mavros apm.launch fcu_url:=“udp://127.0.0.1:14550@14550”

which does not connect to the SITL at all, it just runs the default mavros and waits for an SITL to connect, I guess.

The second one is similar:

roslaunch mavros apm.launch fcu_url:=“udp://127.0.0.1:14560@14560”

which connects to the correct quad, but there is no topic update that one can echo in terminal (e.g. /mavros/global_position/local topic) and after a short while I get the following ROS warning:

[ WARN] [1574759517.528657495]: TM : RTT too high for timesync: 1574759367495.41 ms.

The strange thing is that when I try to arm, change mode of the (second) quad or send position/velocity setpoints via MAVROS, it responds normally, but I can’t see the expected changes in the topics that I wanted to echo in the terminal (can’t echo anything at all).

When I use MAVProxy GCS, it offers 2 UDP output ports for each instance and I can connect MAVROS in that situation, but I do not want to do that because I want to decrease computational overhead.

My questions are:

  1. SITLs are distinctly running, but how can I connect MAVROS to each?
  2. What could be the reason behind the huge RTT time? AirSim utilizes GPU resources (it has its own weird problems as well, but that’s a discussion for another post), so I do not think it would cause such problem and with a GCS the RTT drops to normal levels.
  3. Is there an easier way to achieve what I’m trying to do?

Hope I’ll be understood, sorry for the long post and thanks in advance,

Burak

2

@BurakHocaoglu It seems to be a bit complicated setup, I don’t have quite a lot of experience with ROS but will try and test to see what’s happening.

Most of the things look good, since you’re not using Follow behaviour, maybe try removing the multicast part and see if it changes anything?

Also, are things working properly when using just a single copter? If not, then that would be the first thing to fix

Hope this helps!

3

@BurakHocaoglu I was able to get mult-vehicle simulation with AirSim running with ROS, but it’s all really in the very early stages so things “just work”. Will need to dig deeper to figure out more things but posting here.

First thing is the script, my final script which works right now (just posted the relevant portion) -

# start up main copter in the current directory
$COPTER --model airsim-copter --uartA udpclient:$GCS_IP --defaults $BASE_DEFAULTS &

# Set number of "extra" copters to be simulated, change this for increasing the count
NCOPTERS="1"


# now start another copter to follow the first, using
# a separate directory to keep the eeprom.bin and logs separate
for i in $(seq $NCOPTERS); do
    echo "Starting copter $i"
    mkdir -p copter$i

    SYSID=$(expr $i + 1)
    port=$((14550 + $i * 10))
    echo $port

    cat <<EOF > copter$i/identity.parm
SYSID_THISMAV $SYSID
EOF
    pushd copter$i
    $COPTER --model airsim-copter --uartA udpclient:$GCS_IP:$port --instance $i --defaults $BASE_DEFAULTS,identity.parm &
    popd
done
wait

For launching ROS, launching different launch files seperately didn’t work, probably due to multiple ROS masters, therefore wrote a launch file with different namespaces for each copter.
The launch file-

<?xml version="1.0"?>
<launch>
    <!-- UAV0 -->
    <group ns="uav0">
        <!-- MAVROS and vehicle configs -->
        <arg name="ID" value="1"/>
        <arg name="fcu_url" default="udp://:14550@localhost:14590"/>
        <!-- MAVROS -->
        <include file="$(find mavros)/launch/apm.launch">
            <arg name="fcu_url" value="$(arg fcu_url)"/>
            <arg name="gcs_url" value=""/>
            <arg name="tgt_system" value="$(arg ID)"/>
            <arg name="tgt_component" value="1"/>
        </include>
    </group>
    <!-- UAV1 -->
    <group ns="uav1">
        <!-- MAVROS and vehicle configs -->
        <arg name="ID" value="2"/>
        <arg name="fcu_url" default="udp://:14560@localhost:14581"/>
        <!-- MAVROS -->
        <include file="$(find mavros)/launch/apm.launch">
            <arg name="fcu_url" value="$(arg fcu_url)"/>
            <arg name="gcs_url" value=""/>
            <arg name="tgt_system" value="$(arg ID)"/>
            <arg name="tgt_component" value="1"/>
        </include>
    </group>
</launch>
<!-- to add more UAVs (up to 10):
Increase the id
Change the name space
Set the FCU to default="udp://:14540+id*10@localhost:14550+id"
Set the malink_udp_port to 14560+id) -->

I’ve pushed the files to this branch - https://github.com/rajat2004/ardupilot/tree/airsim-ros-multi
The script in the branch would be quite bigger than the original since it was branched from this PR -

Will be great to have your comments or suggestions!

For running the ROS script, first run libraries/SITL/examples/Airsim/test_multi_ros.sh from the ardupilot root directory.
Then for starting ROS, first go to libraries/SITL/examples/Airsim directory, then from there run
roslaunch multi_uav_ros_sitl.launch

Running MAVProxy causes ROS to stop working, but I think that can be fixed by adding more --uart options in the script and making MAVProxy connect to them.

I’m also getting the RTT messages, need to investigate more on this entire thing.
This page could be very helpful, it’s for PX4 but most of the ROS specific stuff is very useful.
https://dev.px4.io/v1.9.0/en/simulation/multi-vehicle-simulation.html

Hope this helps!

4

Hello there,

I’ve been experimenting with the system above and I have observed the following:

  • When I use sim_vehicle.py, MAVProxy also runs and performs a UDP port multiplexing by offering me UDP ports 14550 and 14551 (for first agent, the remaining will have +10 per id, for each port), so that I can connect MAVROS to one and another GCS to the other. However, the ArduCopter SITL binary, by itself, does not provide this multiplexing and when I connect MAVROS, it does not detect that there is an instance of SITL running, waiting for communication as I don’t see any confirmation on MAVROS side. The weird thing is when I send commands by publishing to MAVROS topics or requesting it’s services, the robot responds, so, the problem should be at the side where SITL sends feedback messages. I think only running SITL does not establish a connection where MAVROS can actually collect updates so that I can rostopic echo them into the terminal.

  • The other thing is your launch file setting does not work within my environment because of the port mismatches, but I think it’s a matter of settings of the working environment. However, when I start SITL with sim_vehicle.py, I realized that with each MAVROS launch, I need to keep tgt_system argument at value 1; otherwise, MAVROS does not detect SITL for some reason.

I might be wrong, I’m open to criticism.

Burak

5

Hello,

Your issue with mavros is simple, you just need to set the stream rate correctly to receive data. Check on discuss or a search engine to get the right command.
tgt_system should match the drone SYSID.

6

The multiplexing info seems correct, though I don’t know much about it. For the binaries, I think if more uart connections are added, it should work. I didn’t need to set the stream_rate as @khancyr suggested when using my script to echo the topic, but might be needed.
I think this is what @khancyr is suggesting - Comunication problem pixhawk, arducopter, mavros 0.32 · Issue #1272 · mavlink/mavros · GitHub

Are you using just the launch file or my modified script also? If just the launch file then I think fixing the ports should be enough. Using both the script and launch file together should work, will probably work more on the whole thing today.
Each sim_vehicle.py launch will launch a new vehicle which would be independent of each other, with each of them having SYSID as 1. Even the binaries, when launched with separate instance values, would keep SYSID as 1, just changes the port numbers, IIRC. Therefore connecting through mavros would need tgt_system as 1 like @khancyr said. For different SYSIDs, would need to set them separately like done in the script

Hope this helps!

7

Thanks @khancyr and @rajat2004 , it seems that my problem is solved. Now I have different tgt_system IDs, and it seems MAVROS topics are working and I don’t have to run a MAVProxy either.

8

@BurakHocaoglu Great!

I have opened a PR for adding these files to the ardupilot repo

@khancyr @BurakHocaoglu Any comments, suggestions would be very much appreciated!

9

Hello again,

I managed to launch multiple SITLs within the same Airsim environment with each of them is connected to a separate MAVROS node simultaneously. I have tested it with 9 arducopters. However, I have a problem that it takes too much time to establish stable MAVROS - SITL connections for the whole system, for following reasons:

  1. When I launch a unique MAVROS node for an SITL right after bulk-launching SITLs with multi-copter.sh, I encounter with an error:

[ERROR] [1575895019.922019315]: udp0: sendto: Invalid argument terminate called after throwing an instance of 'std::system_error' what(): Resource deadlock avoided

then, it takes a few trials (or waiting for OS about 15 seconds to - I guess - allocate the port as a resource in OS level or release it) to finally establish the stable UDP connection. When I checked the port usage with sudo lsof -i -P -n, I realized the actual SITL ports are different than the regular 14550, 14560 and so on. Instead, there were extra connections from the original ports to the regular ports. For example, for 4th SITL instance, in one of my tests with 4 SITL instances, it was like 127.0.0.1:48283 -> 127.0.0.1:14580.

  1. Then, even if MAVROS node starts, it can get stuck in the parameter request loop which doesn’t seem to end at all. A similar problem was issued to MAVROS repo in GitHub (in my case, I did not see any clock skew problems, though). I followed the advice given there and added SERIAL1_BAUD 921 to the identity.parm file generated for each copter to be launched when using multi-copter.sh script. It doesn’t seem to solve the problem at all (well, I don’t think baud rate concept will fit UDP communication mechanism).

What might be the problem? I’m open to any ideas.

Hope, I’m clear. Thanks in advance…

Burak

10

Didn’t go uptill 9, AirSim itself is quite heavy, 9 vehicles would have put your system through the paces, particularly if it’s a single laptop like mine. Would have also been quite repetitive to create the settings and the launch file, I was thinking of looking into ways of automating this, writing a python script should be possible.

So you’ve launched multiple SITL binaries and then starting a separate Mavros node for one on them?
Could you share the command or the launch file used for that?

Each SITL binary will need to have a different port for Mavros to connect to, which is being done here-

I think that’s what you’re referring to? Or am I mistaken and it’s something else?

Yes, even I have faced that problem, happened randomly. Setting the baud rate won’t affect this since it’s not serial. No idea how to tackle this

11

I’m using a desktop which has i7-5820K at 3.3 GHz with 6 cores (12 threads) and GTX 970 4 GB, so it does not put much of an overhead. However, I’m going to add more stuff on top of the MAVROS and SITL, so I can’t guarantee that AirSim will run smoothly at the end (If you suggest Gazebo, based on my experiences with it, I’d say it’s not the best simulator at handling resources and making good use of the underlying hardware).

Basically, there are N SITLs each of which has a unique UDP port number; hence, each vehicle is distinct (it does not have to be 9, it was a stress test that I wanted to see how much my setup can handle; my problems can happen with 2 SITLs as well).

For now, in one terminal, I run:

bash multi-copter.sh [GCS_IP] [N]

And in N distinct terminals, I run:

roslaunch mavros uav.launch fcu_url:=“udp://127.0.0.1: port @ port” tgt_system:=“i”

where i is from 0 to (N - 1) and port is (14550 + i * 10). The launch file is basically a copy of apm.launch .

12

Serial baudrate is no use for IP connection. You can speed up the parameters request by raising the SRX_PARMS parameter (change X by the correct number)

The port number you see on udp on mavros side is correct as it auto assign it. But it shouldn’t struggle to connect. Maybe wait for aisim to be fully launched before launching all you mavros instance.