Communicating across multiple computers with CycloneDDS#

Some setups run Autoware on more than one computer. For example, the Autoware nodes run on one computer and a simulator such as AWSIM runs on a second computer. The ROS 2 nodes on all computers must find each other over the network, and the clocks of the computers must agree.

This page assumes that every computer uses CycloneDDS (rmw_cyclonedds_cpp), the default middleware of Autoware.

Before you start#

Do these steps on every computer:

Apply the DDS settings: the kernel buffer sizes, the ~/cyclonedds.xml file, and the RMW_IMPLEMENTATION and CYCLONEDDS_URI environment variables.
Use the same ROS_DOMAIN_ID on every computer. If you do not set it, ROS 2 uses 0. For details, see About ROS_DOMAIN_ID environment variable.
Connect the computers to the same network and give them addresses in the same subnet. Point clouds and images need a high bandwidth. For example, the multi PC AWSIM test connected the two computers directly with a 10 Gigabit Ethernet link.
If a firewall runs on a computer, allow UDP traffic from the other computers. DDS uses UDP for discovery and for data.

Configuring CycloneDDS#

The ~/cyclonedds.xml file from the DDS settings uses only lo, the loopback interface. Nodes on other computers cannot reach it. To communicate with the other computers, change the Interfaces section of ~/cyclonedds.xml in one of the two ways below.

Automatically determine the network interface (convenient)#

With this setting, CycloneDDS will automatically determine the most suitable network interface to use.

<Interfaces>
  <NetworkInterface autodetermine="true" priority="default" multicast="default" />
</Interfaces>

Manually set the network interface (recommended)#

With this setting, you can manually set the network interface to use.

<Interfaces>
  <NetworkInterface autodetermine="false" name="enp38s0" priority="default" multicast="default" />
</Interfaces>

Warning

Replace enp38s0 with the actual network interface name.

Note

ip -brief address lists the network interfaces and their addresses. The ifconfig command of the net-tools package shows the same information.

The full file with a manually set interface looks like this. Only the NetworkInterface line differs from the file on the DDS settings page:

<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
  <Domain Id="any">
    <General>
      <Interfaces>
        <NetworkInterface autodetermine="false" name="enp38s0" priority="default" multicast="default" />
      </Interfaces>
      <AllowMulticast>default</AllowMulticast>
      <MaxMessageSize>65500B</MaxMessageSize>
    </General>
    <Discovery>
      <ParticipantIndex>none</ParticipantIndex>
    </Discovery>
    <Internal>
      <SocketReceiveBufferSize min="10MB"/>
      <Watermarks>
        <WhcHigh>500kB</WhcHigh>
      </Watermarks>
    </Internal>
  </Domain>
</CycloneDDS>

CycloneDDS reads the file only when a node starts. After you change the file, restart the nodes. Also run ros2 daemon stop, so that the ros2 command line tools use the new file.

Using the Autoware Docker images#

The Autoware Docker images contain a /home/aw/cyclonedds.xml file that uses lo only. To reach other computers from a container, start the container with --net host and mount your own file over it:

docker run --rm -it \
  --net host \
  -v /path/to/your/cyclonedds.xml:/home/aw/cyclonedds.xml \
  ghcr.io/autowarefoundation/autoware:universe-cuda-jazzy

For the other docker run options, see docker/README.md → Usage.

Make sure that the computers find each other#

Make sure that UDP multicast works between the computers. On the first computer, run:
```
ros2 multicast receive
```
On the second computer, run:
```
ros2 multicast send
```
The first computer prints a line such as Received from 192.168.10.2:43210: 'Hello World!'. The operating system selects the interface for this test, not ~/cyclonedds.xml.
Make sure that ROS 2 messages pass through CycloneDDS. Run this command on every computer at the same time:
```
ros2 doctor hello
```
Each computer prints a MULTIMACHINE COMMUNICATION SUMMARY table every second. The Subscribed from: list must contain the host names of all the other computers. Press Ctrl+C to stop the command.

If the computers do not find each other, make sure that:

The ~/cyclonedds.xml file of each computer names the network interface of that computer. The interface names can be different on each computer.
The ROS_DOMAIN_ID and the RMW_IMPLEMENTATION are the same on all computers.
The firewall allows UDP.

Time synchronization#

Autoware nodes compare the time stamps of the messages that they receive. When the nodes run on different computers, the clocks of the computers must agree.

This section uses chrony to synchronize the clocks. In this setup, one computer is the time server and the other computers follow it. The example uses 192.168.10.1 for the server and 192.168.10.2 for a client. Replace them with your addresses.

Install chrony on every computer.
```
sudo apt install chrony
```
On the server, add this line to /etc/chrony/chrony.conf to accept requests from the client. Add one allow line for each client.
```
allow 192.168.10.2
```
If the server has no internet connection, also add local stratum 10. Then the server serves its own clock to the clients.
On each client, comment out the pool lines in /etc/chrony/chrony.conf and add these lines:
```
server 192.168.10.1 iburst minpoll 0 maxpoll 5 maxdelay .05
makestep 0 1
```
The minpoll 0 maxpoll 5 options make the client ask the server every 1 to 32 seconds. The makestep 0 1 line lets chrony correct a large offset in one step at the first update.
Restart chrony on every computer.
```
sudo systemctl restart chrony
```
On each client, make sure that chrony follows the server.
```
chronyc sources
chronyc tracking
```
In the output of chronyc sources, the line of the server starts with ^*. In the output of chronyc tracking, the System time line shows the offset from the server.

In the multi PC AWSIM test, this setup reached a time difference of about 50 µs over a direct 10 Gigabit Ethernet link. The full chrony.conf files of that test are in the discussion. For all options, see the chrony documentation.