Skip to content

Perception component reference implementation design#

Purpose of this document#

This document describes the reference implementation of the Perception component in Autoware 2.0.0. It helps developers and users to understand what the Perception component can do today, and how to use, expand, or add to its features.

Launch files#

autoware_launch/launch/autoware.launch.xml starts the Perception component with two component launch files of autoware_launch:

  • component_perception.launch.xml starts the LiDAR pipeline: the common point cloud preprocessing, obstacle segmentation, the occupancy grid map, and object recognition.
  • component_traffic_light.launch.xml starts traffic light recognition.

Both files include the feature launch files of autoware_perception_launch. The parameter files are in autoware_perception_config. To use a different parameter tree, set the perception_config_pkg launch argument.

Object recognition has one launch file for each sensor setup. component_perception.launch.xml uses object_recognition_lidar.launch.xml, or object_recognition_lidar_ptv3.launch.xml when use_semantic_segmentation_ptv3 is true. A product launch file can include one of the other files:

Launch file in launch/object_recognition/Detectors
object_recognition_lidar.launch.xmlLiDAR DNN detector and LiDAR clustering. This is the default.
object_recognition_lidar_ptv3.launch.xmlLiDAR DNN detector and clustering of the PTv3 semantic segmentation.
object_recognition_lidar_radar.launch.xmlThe PTv3 setup plus the radar faraway detector.
object_recognition_camera_lidar_radar.launch.xmlLiDAR DNN detector, camera-LiDAR fusion clustering, irregular object detector, and radar faraway detector. Camera VRU and camera-only detectors are optional.
object_recognition_camera_lidar_radar_serial.launch.xmlThe same detectors, but object mergers combine them before the tracker. Radar tracks join after the tracker.
object_recognition_camera.launch.xmlCamera-only BEV detector (BEVDet) and an optional camera VRU detector.

The lidar_detection_model launch argument selects the LiDAR DNN detector. Its value is model_type/model_name. The model types are centerpoint (default), transfusion, bevfusion, pointpainting, apollo, and clustering.

Whole architecture#

This diagram shows the architecture of the reference implementation.

Sensing, Localization, and Map send data to the Perception component. In Perception, obstacle segmentation sends the obstacle point cloud to object recognition and to the occupancy grid map. Optionally, the occupancy grid map goes back to obstacle segmentation for outlier filtering. Object recognition runs detection, tracking, and prediction. Optionally, the tracked objects go back to detection for interpolation. Traffic light recognition runs the map-based detector, the fine detector, the classifier, the occlusion predictor, multi-camera fusion, the arbiter with V2X signals, and the crosswalk estimator. It also uses the vector map. Perception sends dynamic objects, obstacle points, the occupancy grid map, and traffic signals to Planning.LocalizationOdometryMapVector, point cloudSensingMapVector mapV2XTraffic signalsPerceptionObject recognitionDetectionLiDAR, camera, radar detectorsTrackingmulti_object_trackerPredictionmap_based_predictionDetected objectsInterpolator (optional)Tracked objectsObstaclesegmentationGround removalOccupancygrid mapBlind spotsOutlier filter(optional)Traffic lightrecognitionMap-based detectorFine detectorClassifierOcclusion predictorMulti-camera fusionArbiter (V2X)Crosswalk estimatorPoint cloud, images, radarPoint cloudPoint cloudImages, point cloudDynamic objectsObstacle pointsOccupancy grid mapTraffic signalsPlanning
The sub-components of the reference implementation and their inputs and outputs.

The Perception component has these sub-components:

  • Obstacle segmentation: Finds the points that come from obstacles that the ego vehicle must avoid. These are dynamic objects and also static obstacles, for example construction cones. By default, a crop box filter and the scan ground filter of autoware_ground_segmentation remove the ground. Two other backends are available: a CUDA version (autoware_ground_segmentation_cuda) and the PTv3 semantic segmentation (autoware_ptv3).
  • Occupancy grid map: Finds blind spots. In a blind spot, no information is available and dynamic objects can jump out. autoware_probabilistic_occupancy_grid_map makes the map from the raw point cloud and the obstacle point cloud. The occupancy_grid_map_method launch argument selects pointcloud_based (default), laserscan_based, or multi_lidar_pointcloud_based.
  • Object recognition: Recognizes dynamic objects around the ego vehicle in the current frame and predicts their future trajectories.
    • Detection: Finds the pose, shape, and class of dynamic objects such as vehicles and pedestrians.
      • Detector: Processes each frame and outputs detected objects.
      • Interpolator: Keeps the detection stable. If the output of a detector stops for a short time, the interpolator uses the tracking output to keep the objects. autoware_detection_by_tracker does this in the serial launch file. In the other launch files, the multi-object tracker keeps its tracks when one detector misses an object.
    • Tracking: Associates the detected objects across frames, gives each object an ID, and estimates its velocity.
    • Prediction: Predicts the trajectories of dynamic objects.
  • Traffic light recognition: Recognizes the colors of traffic lights and the directions of arrow signals.

Outputs to Planning#

OutputTopicMessage type
Dynamic objects/perception/object_recognition/objectsautoware_perception_msgs/msg/PredictedObjects
Obstacle points/perception/obstacle_segmentation/pointcloudsensor_msgs/msg/PointCloud2
Occupancy grid map/perception/occupancy_grid_map/mapnav_msgs/msg/OccupancyGrid
Traffic signals/perception/traffic_light_recognition/traffic_signalsautoware_perception_msgs/msg/TrafficLightGroupArray

Internal interface in the perception component#

  • Obstacle segmentation to object recognition
    • Obstacle point cloud: the point cloud of the current frame without the ground. The pointcloud map filter of object recognition then removes the points that match the point cloud map. LiDAR clustering uses the result.
  • Obstacle segmentation to occupancy grid map
    • Obstacle point cloud: the same point cloud. The occupancy grid map also uses the raw point cloud to find free space.
  • Occupancy grid map to obstacle segmentation and object recognition
    • Occupancy grid map: autoware_occupancy_grid_map_outlier_filter can use the map to remove outlier points, but the launch files of Autoware 2.0.0 do not start it. The object validator of object recognition can also use the map (objects_validation_method:=occupancy_grid).

Traffic light recognition#

For each traffic light camera, the map-based detector makes rough regions of interest, the fine detector refines them, the car and pedestrian classifiers find the signal states, and the occlusion predictor marks hidden signals. Then multi-camera fusion combines the cameras, the arbiter adds the V2X signals, and the crosswalk estimator completes the pedestrian signals and sends the result to Planning.For each traffic light cameraMap-based detectorVector map, routeFine detectorYOLOX on the imageCar classifierPedestrian classifierOcclusion predictorLiDAR point cloudMulti-camera fusionAll camerasArbiterAdds V2X signalsCrosswalk estimatorPedestrian signalsPlanningV2XSignals of each camera
The traffic light recognition pipeline with fine detection.

By default, component_traffic_light.launch.xml runs traffic_light_recognition_fine.launch.xml for the cameras camera6 and camera7. For each camera, these nodes run:

  1. autoware_traffic_light_map_based_detector projects the traffic lights of the vector map on the route into the image, and outputs rough regions of interest (ROIs).
  2. autoware_traffic_light_fine_detector finds the exact traffic light ROIs in the rough ROIs with a YOLOX model.
  3. Two autoware_traffic_light_classifier nodes find the colors and arrows of the car signals and of the pedestrian signals.
  4. autoware_traffic_light_occlusion_predictor uses the LiDAR point cloud to mark the signals that an object hides from the camera.

Then the results of all cameras go through these nodes:

  1. autoware_traffic_light_multi_camera_fusion combines the results of the cameras.
  2. autoware_traffic_light_arbiter combines the camera result with the V2X signals on /perception/traffic_light_recognition/external/traffic_signals.
  3. autoware_crosswalk_traffic_light_estimator estimates the states of pedestrian signals that the cameras cannot see. It publishes the final result.

Two other detection modes are available. With traffic_light_recognition/use_high_accuracy_detection:=false, the classifiers use the rough ROIs directly. With traffic_light_recognition/high_accuracy_detection_type:=whole_image_detection, a YOLOX model finds the traffic lights in the whole image.

Architecture for object recognition#

Nine kinds of detectors send detected objects to the multi-object tracker, each on its own input channel. Base 3D detection and LiDAR clustering run by default. The tracker sends tracked objects to prediction, and prediction sends predicted objects to Planning.Detectors and their tracker channelsIndigo: runs in the default launch fileDetected objectsBase 3D detectionlidar_centerpointNear-object 3D detectionlidar_centerpoint_short_rangeCamera-only 3D detectioncamera_streampetrCamera VRU detectioncamera_vruRadar faraway detectionradarSemantic segmentation clusteringlidar_clusteringLiDAR clusteringlidar_clusteringCamera-LiDAR fusion clusteringcamera_lidar_fusionIrregular object detectioncamera_lidar_fusion_irregularMulti-objecttrackerUp to 12 channelsPredictionmap_based_predictionTrackedobjectsPlanningPredicted objects
The detectors feed the multi-object tracker through separate channels.

Each detector publishes its own DetectedObjects topic. The multi-object tracker takes each topic as one input channel. The tracker combines the channels, so most launch files do not need an object merger. The channel name tells the tracker how much it can trust the channel (see Multi-object tracking).

Base 3D detection#

Autoware primarily uses ML-based methods for 3D detection, referred to as Base 3D Detection. Available methods include:

The detection range depends on the model. The default model parameters cover a square of ±76.8 m around the vehicle for CenterPoint, ±92.16 m for TransFusion, and ±122.4 m for BEVFusion.

If you want to use camera-LiDAR fusion, you can use the camera-LiDAR model of BEVFusion (bevfusion_camera_lidar) or PointPainting (lidar_detection_model:=pointpainting, in the camera-LiDAR launch files). Base 3D Detection is a critical component of the architecture, so it needs stable performance. For this reason, the design recommends against camera-LiDAR fusion methods in environments where the camera data often drops. With these methods, a camera drop affects the base detection:

  • The camera-LiDAR model of BEVFusion skips the detection until each camera sent one image. After that, it uses the last image of each camera, also when the image is old (bevfusion_node.cpp).
  • The fusion node of PointPainting waits for the 2D detections of the cameras until a timer expires. Then it fuses the point cloud with the detections that arrived (autoware_image_projection_based_fusion).

The output topic is /perception/object_recognition/detection/<model_type>/objects. The tracker channel is lidar_<model_type>, for example lidar_centerpoint. If use_validator is true, an object validator of autoware_detected_object_validation removes the objects that have no obstacle points.

Near-object 3D detection#

As optional method for detection to enhance detection of nearby objects, especially pedestrians and cyclists, we have introduced Near-Object 3D Detection. This can serve as a supplementary detection method alongside Base 3D Detection.

To enable it, set use_short_range_detection:=true. Autoware then starts a second CenterPoint node (lidar_short_range_centerpoint) with the short-range model. Its tracker channel is lidar_centerpoint_short_range. The short-range model uses a higher-resolution voxel grid to detect small objects better. In the lidar_centerpoint model bundle v4.1, the short-range model uses 0.16 m voxels and covers a square of ±51.2 m around the vehicle. The base model uses 0.32 m voxels and covers ±76.8 m. The README.md of the model bundle on Hugging Face lists the parameters of each model.

Camera-only 3D detection#

Camera-only 3D detection finds objects that LiDAR-based methods detect badly, for example objects behind trees and objects far away.

  • object_recognition_camera.launch.xml runs BEVDet on six cameras. Its output goes to the tracker channel camera_streampetr.
  • In object_recognition_camera_lidar_radar.launch.xml, set use_camera_only_detection:=true to give the tracker the objects of an external camera-only detector. By default, the topic is /perception/object_recognition/detection/camera_only/objects and the channel is camera_streampetr.
  • Other camera-only detectors are available as packages: StreamPETR and BEVFormer.

The design plans a high-confidence threshold for Camera-Only 3D Detection, to reduce the effect of false positives. Autoware 2.0.0 does not apply such a threshold. BEVDet keeps the objects with a score of at least 0.2 (score_threshold in bevdet.param.yaml of autoware_tensorrt_bevdet). The base CenterPoint model uses 0.35. The tracker can start new tracks from the camera_streampetr channel and trusts its existence probability (input_channels.param.yaml).

The camera VRU detector (use_camera_vru_detector:=true) is a related option. VRU means vulnerable road users, for example pedestrians and cyclists. autoware_image_object_locator makes 3D objects from the 2D boxes of the camera detector. Its tracker channel is camera_vru.

Radar-only faraway object 3D detection#

For enhanced detection of distant objects, we use Radar-Only 3D Detection. The radar launch files filter the radar objects with autoware_object_sorter and the object lanelet filter. They publish the result on /perception/object_recognition/detection/radar/far_objects, and the tracker reads it on the radar channel. For more details, see the document on faraway radar object detection.

3D semantic segmentation#

3D semantic segmentation improves the detection of objects that 3D detectors find badly, especially vegetation and traffic cones. It gives a class label to each point. Then it outputs the non-ground point cloud and the labeled point cloud.

Available methods include:

  • PTv3: set use_semantic_segmentation_ptv3:=true. PTv3 then also replaces the ground removal of obstacle segmentation.
  • FRNet: available as a package. The launch files of Autoware 2.0.0 do not start it.

To make Autoware objects from the labeled points, the label-based Euclidean clustering of autoware_euclidean_cluster groups the points of each label. Its tracker channel is lidar_clustering.

Cluster-based 3D detection#

To enhance detection of objects that LiDAR-based methods detect badly, we offer Cluster-Based 3D Detection. Cluster-Based 3D Detection consists of many nodes. The pipeline is as follows.

The pointcloud map filter removes the points of the obstacle point cloud that match the point cloud map. In LiDAR clustering, Euclidean clustering, a convex hull conversion, and the lanelet filter make objects for the lidar_clustering channel (default LiDAR launch file). In camera-LiDAR fusion (camera-LiDAR launch files), an optional segmentation filter that uses the masks of the 2D detector, Euclidean clustering, ROI cluster fusion with the ROIs of the 2D detector, the low-intensity filter, a convex hull conversion, and the lanelet filter make objects for the camera_lidar_fusion channel. Both go to the multi-object tracker.Obstacle segmentationPoints without the groundPointcloud map filterRemoves points that match the mapPoint cloud mapLiDAR clusteringDefault LiDAR launch fileEuclidean clusteringVoxel grid basedFeature removerConvex hull shapeLanelet filterKeeps objects on the lanesChannel lidar_clusteringCamera-LiDAR fusionCamera-LiDAR launch filesSegmentation filterOptionalEuclidean clusteringVoxel grid basedROI cluster fusionClass from the 2D boxes2D detectorYOLOXLow-intensity filterRaindrop clustersFeature removerConvex hull shapeLanelet filterKeeps objects on the lanesROIsMasksChannel camera_lidar_fusionMulti-object tracker
LiDAR clustering and camera-LiDAR fusion clustering.

Cluster-Based 3D Detection is based on Euclidean clustering of the non-ground LiDAR points:

  • LiDAR clustering (object_recognition_lidar.launch.xml): The clusters become objects with a convex hull shape. The object lanelet filter keeps the objects on the lanes. The tracker channel is lidar_clustering.
  • Camera-LiDAR fusion clustering (camera-LiDAR launch files): ROI cluster fusion gives each cluster the class of the 2D detection (YOLOX) that it falls into. The low-intensity cluster filter of autoware_raindrop_cluster_filter removes clusters from raindrops. An optional image segmentation filter removes points of classes such as the sky before the clustering. The tracker channel is camera_lidar_fusion.
  • Irregular object detector (use_irregular_object_detector, true by default in the camera-LiDAR launch files): ROI point cloud fusion finds small unknown objects in the 2D detections. The tracker channel is camera_lidar_fusion_irregular.

This can be used as supplementary detection alongside Base 3D Detection. In the serial launch file, autoware_detection_by_tracker also uses the clusters to keep the tracked objects, and autoware_object_merger combines the results.

The processing time of this pipeline increases with the number of points and images. For this reason, the design recommends that you do not use this pipeline when the processing time is critical, or that you use it only for a narrow detection range. The default launch files do not narrow the range of the clustering. Its crop box is off by default (use_low_height_cropbox:=false). When it is on, it covers ±200 m in x and y (voxel_grid_based_euclidean_cluster.param.yaml of autoware_perception_config).

Multi-object tracking#

autoware_multi_object_tracker gives each object an ID and estimates its velocity. It works as follows:

Up to 12 detection channels send objects to the input manager. For each batch of objects, the tracker predicts its tracks, associates the objects, updates the matched tracks, prunes old tracks, and spawns new tracks. A timer publishes the tracked objects to prediction.lidar_centerpointlidar_clusteringradarMore channels, up to 12Input managerKeeps a queue for each channel and makes batches of objects in time orderFor each batchPredictTo the object timeAssociateBEV or polar gatesUpdateMatched tracksPruneOld or overlappingSpawnNew tracksTracksVehicle, pedestrian, polygon, staticPublishTimer at 10 HzTracked objectsPrediction
The processing steps of the multi-object tracker.

The key features are as follows.

  • Multiple input channels: The tracker takes up to 12 detection topics. The parameter file input_channels.param.yaml gives each channel flags: whether the channel can start new tracks, and whether the tracker can trust its existence probability, size, class, and orientation. For example, the lidar_clustering channel can start tracks, but the tracker does not trust its size and orientation.
  • No waiting for synchronization: The tracker does not use a message filter to synchronize its inputs. The input manager keeps a queue for each channel. It uses the slowest channel that still publishes as the time reference, and gives the tracker the objects of all channels in time order. If a channel stops, the tracker uses the other channels.
  • Periodic output: With the default parameters, the tracker publishes on a timer at 10 Hz (publish_on_timer, publish_rate). It predicts the tracks to the publish time (delay_compensation).

An earlier version of this design placed a separate priority object merger in front of the tracker. The multi-channel input of the tracker now covers this role. The old object_merger uses approximate time synchronization of two inputs. As a result, it waits for the slower input, and it does not output a frame if one input is missing:

Timing of a synchronized object merger. The data of frame t arrives at the base 3D detector and at the cluster-based detector. The cluster-based detector finishes later. The object merger merges and publishes frame t only after the slower detector finishes. The cluster-based result of frame t+1 is lost, so the merger publishes no output for frame t+1, although the base 3D result of t+1 is there.Timett+1t+2Data acquisitionBase 3D detectionCluster-based detectionObject mergertt+1t+2tt+1 lostt+2tno t+1 outputt+2Processing timeFrame t: merged only after the slower input arrives
A synchronized merger waits for the slower detector, and gives no output when one input is missing.

The multi-channel tracker queues each detection result when it arrives. It updates the tracks when a result of the time reference channel arrives, and a timer publishes the tracks at a fixed rate. A late or missing channel does not stop the output:

Timing of the multi-channel tracker. Each detection result goes into a queue when it arrives. When a result of the slowest channel, the time reference, arrives, the tracker updates its tracks with all queued results in time order. A timer publishes the tracks at a fixed rate. When the cluster-based result of frame t+1 is lost, the timer still publishes the predicted tracks, and the next update also uses the queued base 3D result of frame t+1.Timett+1t+2Data acquisitionBase 3D detectionCluster-based detectionSlowest channel: time referenceTracker updateQueued results, in time orderTracker publishTimer, fixed ratett+1t+2tt+2t+1 losttt+1, t+2Predicted tracksProcessing time
A lost detection result does not stop the output of the multi-channel tracker.

The earlier design also proposed a stationary object detection step to reduce the processing time. The tracker of Autoware 2.0.0 does not have this step. It has a static tracker type, which it uses for objects of the HAZARD class.

Prediction#

autoware_map_based_prediction predicts the paths of the tracked objects from the lanes of the vector map. It publishes /perception/object_recognition/objects. The prediction launch file also supports SIMPL, an ML-based prediction model (prediction_model_type:=simpl). Prediction needs the vector map. Without it (use_vector_map:=false), the launch file does not start prediction.