轻量级激光雷达相机融合模型,实现实时3D动态物体检测与轨迹预测。
An Efficient LiDAR-Camera Fusion Network for Multi-Class 3D Dynamic Object Detection and Trajectory Prediction
- 融合激光雷达与摄像头数据,用Mamba+Transformer实现高效检测。
- 在CODa上检测准确率提升3.71%,行人轨迹预测误差降低0.408米。
- 可在低配显卡上实时运行,适合服务机器人部署。
服务型机器人在执行任务时常需避开动态物体,但计算资源有限。为推动其在复杂动态环境中的实用化,本文提出一种高效的多模态3D感知框架,融合激光雷达与摄像头输入,实现对行人、车辆、骑行者等的实时3D空间检测与轨迹预测。该框架包含两个新模型:1)结合Mamba与Transformer的统一模态检测器(UniMT),兼具高精度与快速推理;2)基于参考轨迹的多类别变换器(RTMCT),支持灵活长度的多样化轨迹预测。在CODa基准测试中,方法在检测(mAP提升3.71%)和轨迹预测(行人minADE₅降低0.408m)上均优于现有方法。在nuScenes检测基准上,检测模型达mAP 72.7%、NDS 75.3%,性能媲美主流激光雷达-相机融合方法。系统展现优异泛化能力与部署潜力——在配备入门级NVIDIA RTX 3060 GPU的轮椅机器人上,实现13.9帧/秒的实时推理且精度满意。代码与ROS推理版本已开源。
原文摘要 · Abstract (English)
Service mobile robots are often required to avoid dynamic objects while performing their tasks, but they usually have only limited computational resources. To further advance the practical application of service robots in complex dynamic environments, we propose an efficient multi-modal framework for 3D object detection and trajectory prediction, which synergistically integrates LiDAR and camera inputs to achieve real-time perception of pedestrians, vehicles, and riders in 3D space.The framework incorporates two novel models: 1) a Unified modality detector with Mamba and Transformer (UniMT) for object detection, which achieves high-accuracy object detection with fast inference speed, and 2) a Reference Trajectory-based Multi-Class Transformer (RTMCT) for efficient and diverse trajectory prediction of multi-class objects with flexible-length trajectories. Evaluations on the CODa benchmark demonstrate that our method outperforms existing ones in both detection (+3.71\% in mAP) and trajectory prediction (-0.408m in minADE$_5$ of pedestrians) metrics. Furthermore, on the challenging nuScenes detection benchmark, our detection model achieves competitive performance among LiDAR-camera fusion methods, with a mAP of 72.7\% and NDS of 75.3\%. Remarkably, the system demonstrates exceptional generalizability and practical deployment potential. When transferred and implemented on a wheelchair robot with an entry-level NVIDIA RTX 3060 GPU, it achieves real-time inference at 13.9 frames per second (FPS) with satisfactory accuracy. To facilitate reproducibility and practical deployment, we release the related code of the method at \href{https://github.com/TossherO/3D_Perception}{https://github.com/TossherO/3D\_Perception} and its ROS inference version at \href{https://github.com/TossherO/ros_packages}{https://github.com/TossherO/ros\_packages}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。