Multi‑Sensor Fusion in Autonomous Driving: Challenges, Prerequisites, and Methods

In early 2024 Pony.ai began a new multi‑sensor perception effort on trucks, accumulating extensive experience with various sensor configurations and scenarios.

Why multi‑sensor fusion? Single sensors each have limitations: cameras lack depth information, have limited field‑of‑view, and are sensitive to lighting; lidar provides accurate 3D points but its effective range is around 150 m, angular resolution is low, and it suffers from noise on reflective surfaces; radar offers reliable distance and velocity but no height information and can generate spurious points. Fusion leverages complementary strengths to extend detection distance and improve robustness.

Prerequisites for sensor fusion

1. Motion compensation & time synchronization – ego‑motion correction, handling motion of other objects during sensor sweep, and precise timestamp alignment (GPS‑based or trigger‑based) are essential to avoid spatial errors.

2. Sensor calibration – extrinsic calibration aligns coordinate frames. For lidar‑to‑lidar, ICP (Iterative Closest Point) is commonly used. Camera‑lidar calibration projects 3D lidar points onto the image using known intrinsics and solves for the extrinsic transform.

3. Field‑of‑view considerations – overlapping and non‑overlapping FoVs affect detection confidence; careful handling of occlusions is required.

Fusion methods

Camera‑lidar fusion – project lidar (x, y, z) points onto the image using calibration parameters, enriching pixels with depth for segmentation or deep‑learning models. When the FoVs differ, mounting the sensors close together helps reduce mismatches.

Another approach projects 2‑D detections back to 3‑D using known camera intrinsics, vehicle height, and ground plane assumptions, yielding 3‑D obstacle estimates.

Radar‑lidar fusion – both provide (x, y) positions in a common Cartesian frame; radar also supplies velocity. Combining them improves distance coverage beyond lidar’s ~150 m limit and helps filter noisy points.

Practical road‑test examples from Pony.ai show how fusion extends reliable detection to 200 m, mitigates sensor‑specific noise, and handles challenging cases such as occlusions and high‑visibility obstacles.

Conclusion Each sensor has unique challenges; effective multi‑sensor fusion compensates these weaknesses, enhancing overall perception accuracy and recall for autonomous vehicles.