用图像纹理辅助点云分割,提升3D场景理解效果
xModel-KD: Cross-modal Knowledge Distillation for 3D Scene Perception using LiDAR

- 通过跨模态对齐,融合2D图像纹理与3D点云几何信息
- 在KITTI数据集上比纯激光雷达方法提升2% mIoU
- 适合缺乏密集标注的3D感知任务,尤其关注数据效率
点云分割是3D场景理解的基础任务,但其发展受限于密集3D标注的高昂成本与耗时。2D图像虽具丰富纹理和外观信息,却缺乏显式深度与几何结构;而3D点云虽能精确表达空间几何,却稀疏且无纹理。单一模态难以获得丰富表征,限制泛化能力。尽管多模态方法结合2D图像与3D点云在分类与检索任务中表现优异,仍依赖大规模标注数据,未充分应用于数据高效的密集预测任务。为此,本文提出xModel-KD框架,通过跨模态知识蒸馏,利用2D纹理与3D几何的互补性,学习统一的逐点表示。设计基于对比学习的跨模态融合编码器,强制多视角下对应2D与3D特征的一致性。结合预训练骨干网络与定向融合策略,有效将图像中的外观线索迁移至几何感知的点特征。实验表明,跨模态融合相较仅使用激光雷达的基线,在KITTI数据集上实现2%的绝对mIoU提升,验证了互补多模态信息对可扩展、低标注依赖3D场景理解的有效性。
原文摘要 · Abstract (English)
Point cloud segmentation is a fundamental task in 3D scene understanding. Its progress is constrained by the high cost and time required for dense 3D annotations, making labeled samples difficult to obtain. Beyond annotation scarcity, different sensing modalities face inherent limitations. 2D images provide rich texture and appearance cues, yet they lack explicit depth and geometric structure. In contrast, 3D point clouds capture accurate spatial geometry but are sparse and contain no texture information. As a result, relying on a single modality restricts the richness of learned representations and weakens generalization. Although recent multi-modal methods that combine 3D point clouds with 2D images have demonstrated strong performance in tasks such as classification and retrieval, they typically depend on large-scale labeled datasets and have not been fully exploited for data-efficient dense prediction. To address these limitations, we propose a novel cross-modal knowledge distillation framework, xModel-KD, for 3D point cloud segmentation. Our method exploits the complementary strengths of 2D texture and 3D geometry by learning unified per-point representations through cross-modal alignment. Specifically, we design a cross-modal fusion encoder trained with a contrastive objective that enforces feature consistency between corresponding 2D and 3D representations across multiple views. By integrating powerful pre-trained backbones with a targeted fusion strategy, the proposed framework effectively transfers appearance cues from images to geometry-aware point features. Experimental results show that cross-modal fusion achieves a 2% absolute improvement in mIoU over a LiDAR-only baseline, demonstrating the benefit of leveraging complementary multi-modal information for scalable and annotation-efficient 3D scene understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。