Task scheduling#

This page describes how a node in Autoware controls when its work runs. It covers timers, callback groups, and executors.

For basic information, see Executors and Using callback groups in the ROS 2 documentation.

Overview#

A node does its work in callbacks:

A timer callback runs at a fixed period.
A subscription callback runs when a message arrives.
A service callback runs when a request arrives.

An executor calls these callbacks. A callback group tells the executor which callbacks can run at the same time.

Run the main processing from a timer#

Many nodes of Autoware, for example in planning, control, and localization, run their main logic at a fixed rate, in a timer callback. The callback takes the newest input messages with polling subscribers, computes the output, and publishes it. The topic message handling guideline explains the polling subscribers.

The autoware_simple_pure_pursuit node of Autoware Core is a short example. Its constructor creates a timer with a period of 30 ms:

timer_ = rclcpp::create_timer(
  this, get_clock(), 30ms, std::bind(&SimplePurePursuitNode::on_timer, this));

The timer callback takes the inputs, calls the algorithm, and publishes the result:

void SimplePurePursuitNode::on_timer()
{
  // 1. Subscribe data
  const auto odom_ptr = odom_sub_.take_data();
  const auto traj_ptr = traj_sub_.take_data();
  if (!odom_ptr || !traj_ptr) {
    return;
  }
  ...
  // 4. Delegate to core logic
  const auto control_command = core_logic_->create_control_command(odom, traj);
  ...
  // 6. Publish control command
  pub_control_command_->publish(control_command);
}

If an input is not available yet, the callback returns and tries again in the next period. To report this, use throttled logging. See Console logging.

Use the clock of the node#

Create a timer with rclcpp::create_timer(this, get_clock(), period, callback). This timer uses the clock of the node. When the use_sim_time parameter is true, the clock of the node follows the /clock topic of a simulator or of ros2 bag play --clock. As a result, the node runs at the correct rate in the simulators and with recorded data.

create_wall_timer() always uses the steady clock and ignores use_sim_time. Use it only for work that must follow the real time, also in a simulation. Most nodes in Autoware Core and Autoware Universe use rclcpp::create_timer().

Timers in nodes that support Agnocast#

autoware::agnocast_wrapper::Node (see Class design) has no create_timer() member. Use the free function autoware::agnocast_wrapper::create_timer() with the same arguments, and store the timer as AUTOWARE_TIMER_PTR. This example comes from autoware_ekf_localizer:

timer_control_ = autoware::agnocast_wrapper::create_timer(
  this, get_clock(), rclcpp::Duration::from_seconds(params_.ekf_dt),
  std::bind(&EKFLocalizerNode::timer_callback, this));

Make the period a parameter#

If the rate of a node can change between vehicles or use cases, read the period from a parameter. For example, autoware_path_generator reads its rate from the planning_hz parameter:

timer_ = rclcpp::create_timer(
  this, get_clock(), rclcpp::Rate(params.planning_hz).period(),
  std::bind(&PathGenerator::run, this));

Add the parameter to the parameter file and to the schema file. See Parameters.

Use callback groups for callbacks that run in parallel#

All callbacks of a node are in the default callback group, unless you give them another group. The default group is mutually exclusive: the executor runs only one of its callbacks at a time, also with a multi-threaded executor. As a result, a long callback delays all the other callbacks of the node.

If a callback must run while another callback is busy, do these steps:

Put the callbacks in different callback groups.
Run the node with a multi-threaded executor. See Select the executor.

For example, autoware_ndt_scan_matcher of Autoware Core creates three mutually exclusive callback groups. The map update timer, the initial pose subscription, and the sensor points subscription each get one:

timer_callback_group_ = this->create_callback_group(rclcpp::CallbackGroupType::MutuallyExclusive);
rclcpp::CallbackGroup::SharedPtr initial_pose_callback_group =
  this->create_callback_group(rclcpp::CallbackGroupType::MutuallyExclusive);
rclcpp::CallbackGroup::SharedPtr sensor_callback_group =
  this->create_callback_group(rclcpp::CallbackGroupType::MutuallyExclusive);

AUTOWARE_SUBSCRIPTION_OPTIONS initial_pose_sub_opt;
initial_pose_sub_opt.callback_group = initial_pose_callback_group;
AUTOWARE_SUBSCRIPTION_OPTIONS sensor_sub_opt;
sensor_sub_opt.callback_group = sensor_callback_group;

The NDT alignment in the sensor points callback can take a long time. Because of the separate groups, the node still receives the initial poses during the alignment.

Follow these rules:

Use rclcpp::CallbackGroupType::MutuallyExclusive groups. Autoware Core and Autoware Universe use them in almost all cases.
Protect the data that callbacks of different groups share, for example with std::mutex. autoware_ndt_scan_matcher keeps such data in a Guarded object, which gives access to the value only while it holds a mutex lock.
A callback group with the second argument of create_callback_group() set to false is not added to the executor of the node. The executor does not run its callbacks. The polling subscribers use such a group. See Prevent calling a callback function.

Do not wait for a response in the group that receives it#

A callback that waits for a service response blocks its callback group until the response arrives. If the response must come through the same callback group, it never arrives, and the node stops. This is a deadlock.

autoware_pose_initializer of Autoware Core avoids it. Its service callback runs in its own callback group, group_srv_. In this callback, the node sends a request to the NDT align service and waits for the response with async_send_request(req).get(). The response arrives in the default callback group, and the node runs with a multi-threaded executor. As a result, the response is not blocked.

For more information, see Using callback groups in the ROS 2 documentation.

Select the executor#

The executor of a standalone node executable is set in the CMakeLists.txt file:

rclcpp_components_register_node() creates an executable with a single-threaded executor. To get a multi-threaded executor, add EXECUTOR MultiThreadedExecutor. autoware_pose_estimator_arbiter of Autoware Universe is an example:
```
rclcpp_components_register_node(${PROJECT_NAME}
  PLUGIN "autoware::pose_estimator_arbiter::PoseEstimatorArbiter"
  EXECUTABLE ${PROJECT_NAME}_node
  EXECUTOR MultiThreadedExecutor
)
```

autoware_agnocast_wrapper_register_node() has two arguments. ROS2_EXECUTOR selects the executor when Agnocast is not used. The default is SingleThreadedExecutor. AGNOCAST_EXECUTOR selects the executor when Agnocast is used. The default is SingleThreadedAgnocastExecutor. autoware_ndt_scan_matcher uses:
```
ROS2_EXECUTOR MultiThreadedExecutor
AGNOCAST_EXECUTOR AgnocastOnlyCallbackIsolatedExecutor
```
An executor whose name starts with AgnocastOnly requires a node that inherits from autoware::agnocast_wrapper::Node. The opposite is also true: when Agnocast is enabled, a node that inherits from autoware::agnocast_wrapper::Node stops at construction unless AGNOCAST_EXECUTOR is an AgnocastOnly executor. Autoware Core and Autoware Universe use AgnocastOnlyCallbackIsolatedExecutor for these nodes. For the list of the executors, see the README of autoware_agnocast_wrapper.

When a launch file loads the node into a component container, the executor of the container runs the node, and the setting in CMakeLists.txt has no effect. component_container has a single-threaded executor, and component_container_mt has a multi-threaded executor. agnocast_env.launch.xml of autoware_agnocast_wrapper selects the container executable from its use_multithread and use_agnocast arguments. The default of use_agnocast is the value of the ENABLE_AGNOCAST environment variable.

When Agnocast is enabled, a node that inherits from autoware::agnocast_wrapper::Node cannot be loaded into a component container. It runs as its own process, with the executor that CMakeLists.txt selects.