用合成数据训练视频深度模型，支持任意长度视频的高精度深度估计。
Depth Any Video with Scalable Synthetic Data
- 构建4万段5秒合成视频，带精确深度标注，解决真实数据稀缺问题。
- 采用混合时长训练和流匹配技术，可处理不同帧率与长度视频，单帧亦可用。
- 支持150帧长序列的高分辨率深度插值，性能优于现有生成式模型。
视频深度估计长期受限于一致且可扩展的真实深度数据稀缺，导致结果不一致且不可靠。本文提出 Depth Any Video 模型，通过两项关键创新应对该挑战：首先，开发可扩展的合成数据流水线，从多样虚拟环境中实时捕获视频深度数据，生成40,000段5秒视频，每段均含精确深度标注；其次，利用生成式视频扩散模型的强大先验，结合旋转位置编码与流匹配等先进技术，有效处理真实世界视频，提升灵活性与效率。不同于以往固定时长视频模型，本方法引入新型混合时长训练策略，可处理不同长度视频并适应多种帧率，甚至支持单帧输入。推理阶段提出一种深度插值方法，使模型可在长达150帧的序列上推断出高分辨率视频深度。实验表明，该模型在空间精度与时间一致性上均优于所有先前生成式深度模型。代码与模型权重已开源。
原文摘要 · Abstract (English)
Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse virtual environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates-even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency. The code and model weights are open-sourced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。