用统一模型实现视频几何的精准连续估计
Towards Consistent Video Geometry Estimation

- 基于纯Transformer架构,动态分块注意力适应不同推理模式
- 在多个数据集上达到当前最佳,支持长视频与流式处理
- 可同时预测深度、法向量和点云,适合三维视觉任务
本文提出ViGeo,一种前馈式基础模型,用于从视频序列中恢复空间密集且时间一致的几何信息。该模型采用无任务特化修改的普通Transformer架构,支持流式、全序列和长视频推理。核心设计为动态分块注意力,在训练时同时引入双向和单向时间上下文,测试时无需重训练即可自适应调整注意力模式。为提升监督质量,进一步提出基于补全的数据精炼框架:训练一个视频深度补全教师模型,利用稀疏噪声标注和视频/多视角上下文生成稠密、时间连贯且几何可靠的训练目标。除深度图和点云外,ViGeo还在同一框架下预测表面法向量。仅使用公开数据集训练,其在在线、离线及长视频深度估计、表面法向量估计和视频点云估计任务中均达到当前最优性能。
原文摘要 · Abstract (English)
This work presents ViGeo, a feed-forward foundation model for recovering spatially dense and temporally consistent geometry from video sequences. Built upon a plain transformer architecture without task-specific architectural modifications, ViGeo supports streaming, full-sequence, and long-video inference within a unified model. The key design is dynamic chunking attention, which exposes the model to both bidirectional and causal temporal contexts during training and allows it to adapt its attention pattern at test time without retraining. To improve supervision quality, we further introduce a completion-based data refinement framework. This framework trains a video depth completion teacher that conditions on sparse and noisy annotations and exploits video/multi-view context to produce dense, temporally coherent, and geometrically reliable training targets. Beyond depth and point maps, ViGeo also predicts surface normals within the same framework. Trained solely on public datasets, ViGeo achieves state-of-the-art performance across online, offline, and long-video depth estimation, surface normal estimation, and video point map estimation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。