Defining temporal performance metrics on components#

Motivation to defining temporal performance metrics#

Objective of the page#

This page introduces policies to define metrics to evaluate temporal performance on components of Autoware. The term "temporal performance" is often used throughout the page in order to distinguish between functional performance, which referred to as accuracy as well, and time-related performance.

It is expected that most algorithms employed for Autoware are executed with as high frequency and short response time as possible. In order to achieve safe autonomous driving, one of the desired outcomes is no time gap between perceived and actual situation. The time gap is commonly referred to as delay. If the delay is significant, the system may determine trajectory and maneuver based on outdated situation. Consequently, if the actual situation differs from the perceived one due to the delay, the system may make unexpected decisions.

As mentioned above, this page presents the policies to define metrics. Besides, the page contains lists of sample metrics that are crucial for the main functionalities of Autoware: Localization, Perception, Planning, and Control.

Contribution of the temporal performance metrics#

Temporal performance metrics are important for evaluating Autoware. These metrics are particularly useful for assessing delays caused by new algorithms and logic. They can be employed when comparing the temporal performance of software on a desktop computer with that on a vehicle during the vehicle integration phase.

In addition, these metrics are useful for designers and evaluators of middleware, operating systems, and computers. They are selected based on user and product requirements. One of these requirements is to provide sufficient temporal performance for executing Autoware. "Sufficient temporal performance" is defined as a temporal performance requirement, but it can be challenging to define the requirement because it varies depending on the product type, Operational Design Domain (ODD), and other factors. Then, this page specifically focuses on temporal performance metrics rather than requirements.

Temporal performance metrics are important for evaluating the reliability of Autoware. However, ensuring the reliability of Autoware requires consideration of not only temporal performance metrics but also other metrics.

Tools for evaluating the metrics#

There are several tools available for evaluating Autoware according to the metrics listed in the page. For example, both CARET and ros2_tracing are recommended options when evaluating Autoware on Linux and ROS 2. If you want to measure the metrics with either of these tools, refer to the corresponding user guide for instructions. It's important to note that if you import Autoware to a platform other than Linux and ROS 2, you will need to choose a supported tool for evaluation.

The page does not aim to provide instructions on how to use these tools or measure the metrics. Its primary focus is on the metrics themselves, as they are more important than the specific tools used. These metrics retain their relevance regardless of the employed platform.

Policies to define temporal performance metrics#

As mentioned above, the configuration of Autoware varies by the product type, ODD, and other factors. The variety of configurations makes it difficult to define the uniform metrics for evaluating Autoware.
However, the policies used to define them are basically reused even when the configuration changes. Each of temporal performance metrics is categorized into two types: execution frequency and response time. Although there are many types of metrics, such as communication latency, the only two types are considered for simplicity.
Execution frequency is observed using rate of Inter-Process Communication (IPC) messages. You will find an enormous number of messages in Autoware, but you don't have to take care of all. Some messages might be critical to functionality and they should be chosen for evaluation.
Response time is duration elapsed through a series of processing. A series of processing is referred to as a path. Response time is calculated from timestamps of start and end of a path. Although many paths can be defined in Autoware, you have to choose significant paths.

As a hint, here are some characteristics of message and path in order to choose metrics.

1. Messages and paths on boundaries where observed values from sensors are consumed
2. Messages and paths on boundaries of functions, e.g., a boundary of perception and planning
3. Messages and paths on boundaries where timer-based frequency is switched
4. Messages and paths on boundaries where two different messages are synchronized and merged
5. Messages that must be transmitted at expected frequency, e.g., vehicle command messages

Those hints would be helpful for most configurations but there may be exclusions. Defining metrics precisely requires an understanding of configuration.

In addition, it is recommended that metrics be determined incrementally from the architectural level to the detailed design and implementation level. Mixing metrics at different levels of granularity can be confusing.

List of sample metrics#

This section demonstrates how to define metrics according to the policies explained and has lists of the metrics for Autoware launched according to the tutorial. The section is divided into multiple subsections, each containing a model diagram and an accompanying list that explains the important temporal performance metrics. Each model is equipped with checkpoints that serve as indicators for these metrics.

The first subsection presents the top-level temporal performance metrics, which are depicted in the abstract structure of Autoware as a whole. The detailed metrics are not included in the model as they would add complexity to it. Instead, the subsequent section introduces the detailed metrics. The detailed metrics are subject to more frequent updates compared to the top-level ones, which is another reason for categorizing them separately.

Each list includes a column for the reference value. The reference value represents the observed value of each metric when Autoware is running according to the tutorial. It is important to note that the reference value is not a required value, meaning that Autoware does not necessarily fail in the tutorial execution if certain metrics do not fulfill the reference value.

Top-level temporal performance metrics for Autoware#

The diagram below introduces the model for top-level temporal performance metrics.

The following three policies assist in selecting the top-level performance metrics:

• Splitting Autoware based on components that consume observed values, such as sensor data, and considering the processing frequency and response time around these components
• Dividing Autoware based on the entry point of Planning and Control and considering the processing frequency and response time around these components
• Showing the minimum metrics for the Vehicle Interface, as they may vary depending on the target vehicle

Additionally, it is assumed that algorithms are implemented as multiple nodes and function as a pipeline processing system.

Detailed temporal performance metrics for Perception#

The diagram below introduces the model for temporal performance metrics for Perception.

The following two policies assist in selecting the performance metrics:

• Regarding the frequency and response time at which Recognition results from Object Recognition and Traffic Light Recognition are consumed in Planning
• Splitting Perception component on merging points of data from multiple processing paths and considering the frequency and response time around that point

The following list shows the temporal performance metrics for Perception.

Detailed temporal performance metrics for paths between obstacle segmentation and planning#

Obstacle segmentation, which is a crucial part of Perception, transmits data to Planning. The figure below illustrates the model that takes into account performance metrics related to Obstacle segmentation and Planning.

In Autoware 2.0.0, the occupancy grid map uses the output of obstacle segmentation and the raw pointcloud from Sensing. The model shows this order. The checkpoint numbers stay the same as in the tables.

The following list shows the temporal performance metrics around Obstacle segmentation and Planning.