arXiv:2501.06550cs.CV2025-01中稿 · Information Fusion…被引 16

解决激光雷达与摄像头融合中的特征错位和任务抑制问题,提升3D目标检测精度。

CoreNet: Conflict Resolution Network for Point-Pixel Misalignment and Sub-Task Suppression of 3D LiDAR-Camera Object Detection

  • 设计双流转换模块,实现图像到世界空间的唯一映射,缓解点像素错位。
  • 引入任务专用预测器,分离分类与框回归,避免子任务相互干扰。
  • 在nuScenes上达到75.6% NDS,无需测试时增强或模型集成,适合多模态感知系统研发。

融合不同传感器的多模态输入是提升3D目标检测性能的有效方法。然而,现有方法忽略了两个关键冲突:点像素错位和子任务抑制。前者指图像中某个像素特征被投影到同一光线上多个点特征,后者指分类预测与边界框回归之间存在相互抑制。本文提出冲突解析网络(CoreNet)以解决上述问题。首先设计双流转换模块,包含基于光束和基于点的2D到鸟瞰图(BEV)变换,均实现从图像空间到世界空间的近似唯一映射。此外,引入任务专用预测器,采用双分支结构,分别使用类别相关查询和框相关查询处理子任务。每个任务查询由任务特有特征与通用特征构成,使头部能根据任务自适应选择关注信息。在大规模nuScenes数据集上的实验表明,CoreNet在不使用测试时增强和模型集成的情况下,达到75.6% NDS和73.3% mAP。充分的消融实验验证了各组件的有效性。代码已开源。

原文摘要 · Abstract (English)

Fusing multi-modality inputs from different sensors is an effective way to improve the performance of 3D object detection. However, current methods overlook two important conflicts: point-pixel misalignment and sub-task suppression. The former means a pixel feature from the opaque object is projected to multiple point features of the same ray in the world space, and the latter means the classification prediction and bounding box regression may cause mutual suppression. In this paper, we propose a novel method named Conflict Resolution Network (CoreNet) to address the aforementioned issues. Specifically, we first propose a dual-stream transformation module to tackle point-pixel misalignment. It consists of ray-based and point-based 2D-to-BEV transformations. Both of them achieve approximately unique mapping from the image space to the world space. Moreover, we introduce a task-specific predictor to tackle sub-task suppression. It uses the dual-branch structure which adopts class-specific query and Bbox-specific query to corresponding sub-tasks. Each task-specific query is constructed of task-specific feature and general feature, which allows the heads to adaptively select information of interest based on different sub-tasks. Experiments on the large-scale nuScenes dataset demonstrate the superiority of our proposed CoreNet, by achieving 75.6\% NDS and 73.3\% mAP on the nuScenes test set without test-time augmentation and model ensemble techniques. The ample ablation study also demonstrates the effectiveness of each component. The code is released on https://github.com/liyih/CoreNet.

3D检测多模态融合激光雷达目标检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。