对比真实与模拟激光雷达数据，发现密度感知距离最能反映几何差异。
Comprehensive Assessment of LiDAR Evaluation Metrics: A Comparative Study Using Simulated and Real Data
- 采用多种噪声、密度等条件测试评估指标，筛选出最优方法。
- 模拟与真实扫描在语义分割上mIoU达21%，几何距离平均为0.63。
- 密度感知距离与模型感知结果相关性最强，适合自动驾驶验证。
为确保自动驾驶系统安全，需在部署前进行严格测试。由于传统物理测试成本高且存在安全隐患，虚拟测试环境（VTE）成为替代方案。通过比较VTE生成的传感器数据与真实数据的相似性，可评估VTE的真实性。本文系统研究了多种用于比较真实与模拟激光雷达扫描的评估指标，在不同噪声、密度、畸变、传感器朝向和通道设置下测试其敏感性和准确性。结果显示，密度感知切尔姆夫距离（DCD）在所有情况下表现最佳。进一步地，基于真实激光雷达数据构建了虚拟测试环境，使用搭载激光雷达、IMU和相机的实验车辆在静态场景中采集数据。在相同位姿下生成模拟激光雷达扫描，并从模型感知和几何相似性两方面进行对比。实际与模拟扫描的语义分割结果相近，修正强度后mIoU为21%，平均密度感知切尔姆夫距离（DCD）为0.63。表明两者几何特性略有差异，但模型输出差异显著。其中，密度感知切尔姆夫距离与感知方法的相关性最高。
原文摘要 · Abstract (English)
For developing safe Autonomous Driving Systems (ADS), rigorous testing is required before they are deemed safe for road deployments. Since comprehensive conventional physical testing is impractical due to cost and safety concerns, Virtual Testing Environments (VTE) can be adopted as an alternative. Comparing VTE-generated sensor outputs against their real-world analogues can be a strong indication that the VTE accurately represents reality. Correspondingly, this work explores a comprehensive experimental approach to finding evaluation metrics suitable for comparing real-world and simulated LiDAR scans. The metrics were tested in terms of sensitivity and accuracy with different noise, density, distortion, sensor orientation, and channel settings. From comparing the metrics, we found that Density Aware Chamfer Distance (DCD) works best across all cases. In the second step of the research, a Virtual Testing Environment was generated using real LiDAR scan data. The data was collected in a controlled environment with only static objects using an instrumented vehicle equipped with LiDAR, IMU and cameras. Simulated LiDAR scans were generated from the VTEs using the same pose as real LiDAR scans. The simulated and LiDAR scans were compared in terms of model perception and geometric similarity. Actual and simulated LiDAR scans have a similar semantic segmentation output with a mIoU of 21\% with corrected intensity and an average density aware chamfer distance (DCD) of 0.63. This indicates a slight difference in the geometric properties of simulated and real LiDAR scans and a significant difference between model outputs. During the comparison, density-aware chamfer distance was found to be the most correlated among the metrics with perception methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。