arXiv:2512.20217cs.CV2025-12被引 1

用极少改动让纯视觉3D检测器学会融合激光雷达,提升鲁棒性。

LiteFusion: Taming 3D Object Detectors from Vision-Based to Multi-Modal with Minimal Adaptation

  • 将激光雷达作为几何信息补丁融入图像特征,不依赖专用编码器
  • 在nuScenes上使纯视觉检测器的mAP提升20.4%,参数仅增1.1%
  • 无需激光雷达输入也能保持性能,适合部署在NPU/FPGA等设备

3D目标检测对智能交通系统的安全与鲁棒性至关重要。现有多模态检测方法通常依赖复杂架构和训练策略以提升精度,但严重依赖激光雷达,在激光雷达缺失时性能显著下降,影响实际应用中的安全性。此外,由于依赖3D稀疏卷积算子(主要优化于NVIDIA GPU),现有方法难以在NPU、FPGA等异构硬件上部署。为此,本文重新审视相机-激光雷达融合范式中激光雷达的作用,提出轻量级多模态3D检测器LiteFusion。不同于将激光雷达点云视为独立模态并使用专用主干网络,LiteFusion将激光雷达数据作为几何信息补充,通过四元数空间融合至图像特征中,保留正交约束,有效建模跨模态域内关系,生成紧凑的跨模态嵌入。在nuScenes数据集上的实验表明,该方法在不使用专用激光雷达编码器的前提下,使基准视觉检测器的mAP提升20.4%、NDS提升19.7%,参数仅增加1.1%。值得注意的是,即使在无激光雷达输入情况下,模型仍保持优异性能,展现出良好的鲁棒性与泛化能力。

原文摘要 · Abstract (English)

3D object detection is fundamental for safe and robust intelligent transportation systems. Current multi-modal 3D object detectors often rely on complex architectures and training strategies to achieve higher detection accuracy. However, these methods heavily rely on the LiDAR sensor so that they suffer from large performance drops when LiDAR is absent, which compromises the robustness and safety of autonomous systems in practical scenarios. Moreover, existing multi-modal detectors face difficulties in deployment on diverse hardware platforms, such as NPUs and FPGAs, due to their reliance on 3D sparse convolution operators, which are primarily optimized for NVIDIA GPUs. To address these challenges, we reconsider the role of LiDAR in the camera-LiDAR fusion paradigm and introduce a novel multi-modal 3D detector, LiteFusion. Instead of treating LiDAR point clouds as an independent modality with a separate feature extraction backbone, LiteFusion utilizes LiDAR data as a complementary source of geometric information to enhance camera-based detection. This straightforward approach completely eliminates the reliance on a 3D backbone, making the method highly deployment-friendly. Specifically, LiteFusion integrates complementary features from LiDAR points into image features within a quaternion space, where the orthogonal constraints are well-preserved during network training. This helps model domain-specific relations across modalities, yielding a compact cross-modal embedding. Experiments on the nuScenes dataset show that LiteFusion improves the baseline vision-based detector by +20.4% mAP and +19.7% NDS with a minimal increase in parameters (1.1%) without using dedicated LiDAR encoders. Notably, even in the absence of LiDAR input, LiteFusion maintains strong results , highlighting its favorable robustness and effectiveness across diverse fusion paradigms and deployment scenarios.

3D检测多模态融合轻量化部署友好

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。