多流网络融合激光雷达与摄像头数据，提升户外3D目标检测精度。
Multistream Network for LiDAR and Camera-based 3D Object Detection in Outdoor Scenes
- 设计三流结构，分别提取点云、高度图和多模态特征。
- 在KITTI基准上达到新纪录，小目标检测提升显著。
- 适合自动驾驶中需高精度感知的场景使用。
激光雷达与可见光图像的融合有望提升户外3D目标检测的准确性。为应对真实场景下的挑战，该领域逐渐受到关注。然而，如何有效融合多模态数据以实现精确检测仍是一个开放问题。为此，本文提出多流检测网络（MuStD），从两种模态中精细提取任务相关特征。网络采用三流结构：LiDAR-PillarNet 流从点云中提取稀疏2D柱状特征；LiDAR-Height Compression 流生成鸟瞰图特征；3D多模态流通过UV映射与极坐标索引融合RGB与激光雷达特征。最终，包含完整空间、纹理与几何信息的特征被精心融合并输入检测头完成3D物体检测。在具有挑战性的KITTI目标检测基准上，通过公开测试服务器评估，本方法在多个类别中达到或超越当前最优性能，同时保持高效性。代码将发布于 https://github.com/IbrahimUWA/MuStD.git。
原文摘要 · Abstract (English)
Fusion of LiDAR and RGB data has the potential to enhance outdoor 3D object detection accuracy. To address real-world challenges in outdoor 3D object detection, fusion of LiDAR and RGB input has started gaining traction. However, effective integration of these modalities for precise object detection task still remains a largely open problem. To address that, we propose a MultiStream Detection (MuStD) network, that meticulously extracts task-relevant information from both data modalities. The network follows a three-stream structure. Its LiDAR-PillarNet stream extracts sparse 2D pillar features from the LiDAR input while the LiDAR-Height Compression stream computes Bird's-Eye View features. An additional 3D Multimodal stream combines RGB and LiDAR features using UV mapping and polar coordinate indexing. Eventually, the features containing comprehensive spatial, textural and geometric information are carefully fused and fed to a detection head for 3D object detection. Our extensive evaluation on the challenging KITTI Object Detection Benchmark using public testing server at https://www.cvlibs.net/datasets/kitti/eval_object_detail.php?&result=d162ec699d6992040e34314d19ab7f5c217075e0 establishes the efficacy of our method by achieving new state-of-the-art or highly competitive results in different categories while remaining among the most efficient methods. Our code will be released through MuStD GitHub repository at https://github.com/IbrahimUWA/MuStD.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。