用生成视频模型补全图像间缺失帧,提升少重叠场景的姿势估计效果。
Can Generative Video Models Help Pose Estimation?
- 通过生成模型补全两图间的中间帧,构建视觉过渡流
- 在4个数据集上超越DUSt3R,跨3个生成模型保持稳定提升
- 适合处理低重叠、无对应点的复杂场景,尤其适合视觉重建任务
从重叠极少或无重叠的图像对中进行姿态估计是计算机视觉中的开放难题。现有方法即使在大规模数据集上训练,仍因缺乏可辨识对应点或视觉重叠而表现不佳。受人类从多样场景推断空间关系能力启发,我们提出InterPose方法,利用预训练生成视频模型中蕴含的丰富先验知识。通过视频模型在两张输入图像间生成中间帧,形成密集视觉过渡,显著简化姿态估计问题。针对当前视频模型可能产生不合理的运动或几何不一致的问题,我们引入自一致性评分,评估采样视频中姿态预测的一致性。实验表明,该方法在三个先进视频模型上均具泛化能力,并在涵盖室内、室外及物体中心场景的四个数据集上持续优于当前最优的DUSt3R。结果表明,利用大规模视频数据训练的生成模型,可为姿态估计提供有效补充,其数据获取远比三维数据更便捷。详情见项目页:https://inter-pose.github.io/
原文摘要 · Abstract (English)
Pairwise pose estimation from images with little or no overlap is an open challenge in computer vision. Existing methods, even those trained on large-scale datasets, struggle in these scenarios due to the lack of identifiable correspondences or visual overlap. Inspired by the human ability to infer spatial relationships from diverse scenes, we propose a novel approach, InterPose, that leverages the rich priors encoded within pre-trained generative video models. We propose to use a video model to hallucinate intermediate frames between two input images, effectively creating a dense, visual transition, which significantly simplifies the problem of pose estimation. Since current video models can still produce implausible motion or inconsistent geometry, we introduce a self-consistency score that evaluates the consistency of pose predictions from sampled videos. We demonstrate that our approach generalizes among three state-of-the-art video models and show consistent improvements over the state-of-the-art DUSt3R on four diverse datasets encompassing indoor, outdoor, and object-centric scenes. Our findings suggest a promising avenue for improving pose estimation models by leveraging large generative models trained on vast amounts of video data, which is more readily available than 3D data. See our project page for results: https://inter-pose.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。