无需3D标注训练神经网格模型，实现高精度零样本3D姿态估计
DINeMo: Learning Neural Mesh Models with no 3D Annotations
- 利用视觉大模型生成伪对应点，实现无3D标注的双向特征匹配
- 在汽车数据集上零样本性能超越此前方法，与全监督模型差距缩小67.3%
- 可高效扩展，加入更多无标注图像即可持续提升性能
类别级3D/6D姿态估计是实现全面3D场景理解的关键步骤，有助于推动机器人与具身智能应用。近期工作采用神经网格模型，从分析-合成视角解决2D和3D任务。尽管对遮挡和域偏移具有更强鲁棒性，但这些方法严重依赖3D标注进行部件对比学习，限制了类别覆盖范围并阻碍规模化。本文提出DINeMo，一种无需3D标注的新型神经网格模型，通过大规模视觉基础模型获取伪对应关系进行训练。采用双向伪对应生成机制，融合局部外观特征与全局上下文信息。在汽车数据集上的实验表明，DINeMo在零样本3D姿态估计中显著优于以往方法，与全监督方法的差距缩小67.3%。该模型在训练中引入更多无标注图像时仍能有效扩展，展现出对依赖3D标注的监督学习方法的优势。
原文摘要 · Abstract (English)
Category-level 3D/6D pose estimation is a crucial step towards comprehensive 3D scene understanding, which would enable a broad range of applications in robotics and embodied AI. Recent works explored neural mesh models that approach a range of 2D and 3D tasks from an analysis-by-synthesis perspective. Despite the largely enhanced robustness to partial occlusion and domain shifts, these methods depended heavily on 3D annotations for part-contrastive learning, which confines them to a narrow set of categories and hinders efficient scaling. In this work, we present DINeMo, a novel neural mesh model that is trained with no 3D annotations by leveraging pseudo-correspondence obtained from large visual foundation models. We adopt a bidirectional pseudo-correspondence generation method, which produce pseudo correspondence utilize both local appearance features and global context information. Experimental results on car datasets demonstrate that our DINeMo outperforms previous zero- and few-shot 3D pose estimation by a wide margin, narrowing the gap with fully-supervised methods by 67.3%. Our DINeMo also scales effectively and efficiently when incorporating more unlabeled images during training, which demonstrate the advantages over supervised learning methods that rely on 3D annotations. Our project page is available at https://analysis-by-synthesis.github.io/DINeMo/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。