提升ViT模型3D感知能力，仅用单物体一次微调就显著增强对应关系理解。
Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuning
- 基于3D对应关系设计微调策略，增强模型跨视角一致性
- 单物体一次微调即带来显著性能提升，适用于姿态估计等任务
- 方法轻量高效，适合已有视觉模型快速注入3D理解能力
视觉基础模型，尤其是ViT系列，已显著提升图像理解能力，但其对3D空间关系的把握仍不清晰。本文系统评估了基于ViT的模型学习3D等变特征的能力，重点考察不同视角下语义嵌入的一致性。结果表明，增强3D等变性可显著提升姿态估计、跟踪和语义迁移等下游任务表现。基于此，我们提出一种基于3D对应关系的简单有效微调策略，显著增强现有视觉模型的3D对应理解能力。令人惊喜的是，仅对单个物体进行一次微调，即可取得显著性能提升。代码已开源：https://github.com/qq456cvb/3DCorrEnhance。
原文摘要 · Abstract (English)
Vision foundation models, particularly the ViT family, have revolutionized image understanding by providing rich semantic features. However, despite their success in 2D comprehension, their abilities on grasping 3D spatial relationships are still unclear. In this work, we evaluate and enhance the 3D awareness of ViT-based models. We begin by systematically assessing their ability to learn 3D equivariant features, specifically examining the consistency of semantic embeddings across different viewpoints. Our findings indicate that improved 3D equivariance leads to better performance on various downstream tasks, including pose estimation, tracking, and semantic transfer. Building on this insight, we propose a simple yet effective finetuning strategy based on 3D correspondences, which significantly enhances the 3D correspondence understanding of existing vision models. Remarkably, finetuning on a single object for one iteration results in substantial gains. Our code is available at https://github.com/qq456cvb/3DCorrEnhance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。