首个统一2D与3D几何匹配的共享权重模型，提升跨模态对应精度。
UniCorrn: Unified Correspondence Transformer Across 2D and 3D
- 采用双流解码器分离外观与位置特征，支持异构模态灵活匹配
- 在7Scenes上2D-3D匹配提升8%，3DLoMatch上3D-3D匹配提升10%
- 共享编码器解码器结构，可端到端训练多任务数据
图像间（2D-2D）、图像与点云间（2D-3D）、点云间（3D-3D）的视觉对应关系是众多3D视觉任务的基础。尽管问题结构相似，现有方法采用针对不同模态组合的专用设计和独立模型。我们提出UniCorrn，首个实现三类任务统一建模的共享权重对应模型。核心洞察是Transformer注意力天然捕捉跨模态特征相似性。设计双流解码器，保持外观与位置特征流分离，支持可堆叠层的端到端学习，并实现跨异构模态的查询式对应估计。架构采用模态特定主干网络，后接共享编码器与解码器，联合训练包含深度图伪点云与真实3D对应标注的多样化数据。UniCorrn在2D-2D匹配上表现优异，在7Scenes（2D-3D）上注册召回率超越先前最优8%，在3DLoMatch（3D-3D）上提升10%。
原文摘要 · Abstract (English)
Visual correspondence across image-to-image (2D-2D), image-to-point cloud (2D-3D), and point cloud-to-point cloud (3D-3D) geometric matching forms the foundation for numerous 3D vision tasks. Despite sharing a similar problem structure, current methods use task-specific designs with separate models for each modality combination. We present UniCorrn, the first correspondence model with shared weights that unifies geometric matching across all three tasks. Our key insight is that Transformer attention naturally captures cross-modal feature similarity. We propose a dual-stream decoder that maintains separate appearance and positional feature streams. This design enables end-to-end learning through stack-able layers while supporting flexible query-based correspondence estimation across heterogeneous modalities. Our architecture employs modality-specific backbones followed by shared encoder and decoder components, trained jointly on diverse data combining pseudo point clouds from depth maps with real 3D correspondence annotations. UniCorrn achieves competitive performance on 2D-2D matching and surpasses prior state-of-the-art by 8% on 7Scenes (2D-3D) and 10% on 3DLoMatch (3D-3D) in registration recall. Project website: https://neu-vi.github.io/UniCorrn
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。