解决超长视频深度估计的时序不一致问题,实现高质量实时推理。
Video Depth Anything: Consistent Depth Estimation for Super-Long Videos

- 用时空融合头替代原模型头部,提升视频时序一致性。
- 在多视频基准上实现零样本新最佳性能,支持数分钟级长视频。
- 支持多种规模模型,最小版本可达30帧/秒实时运行。
Depth Anything 在单目深度估计中表现出色,具备强泛化能力,但在视频中存在时序不一致问题,限制了实际应用。现有方法依赖视频生成模型或光流、相机位姿等几何先验,仅适用于短视频(<10秒),且需在质量和效率间权衡。本文提出 Video Depth Anything,实现超长视频(数分钟以上)的高质量、一致深度估计,同时保持高效性。基于 Depth Anything V2,替换其头部为高效的时空头,并设计一种仅通过约束时间深度梯度的简单有效时序一致性损失,无需额外几何先验。模型在视频深度与无标签图像联合数据集上训练,类似 Depth Anything V2。此外,提出关键帧策略以支持长视频推理。实验表明,本方法可处理任意长度视频,兼顾质量、一致性与泛化能力。多个视频基准上的综合评估显示,该方法在零样本视频深度估计上达到新最优。提供不同规模模型以适应多种场景,最小模型可在30帧/秒下实时运行。
原文摘要 · Abstract (English)
Depth Anything has achieved remarkable success in monocular depth estimation with strong generalization ability. However, it suffers from temporal inconsistency in videos, hindering its practical applications. Various methods have been proposed to alleviate this issue by leveraging video generation models or introducing priors from optical flow and camera poses. Nonetheless, these methods are only applicable to short videos (< 10 seconds) and require a trade-off between quality and computational efficiency. We propose Video Depth Anything for high-quality, consistent depth estimation in super-long videos (over several minutes) without sacrificing efficiency. We base our model on Depth Anything V2 and replace its head with an efficient spatial-temporal head. We design a straightforward yet effective temporal consistency loss by constraining the temporal depth gradient, eliminating the need for additional geometric priors. The model is trained on a joint dataset of video depth and unlabeled images, similar to Depth Anything V2. Moreover, a novel key-frame-based strategy is developed for long video inference. Experiments show that our model can be applied to arbitrarily long videos without compromising quality, consistency, or generalization ability. Comprehensive evaluations on multiple video benchmarks demonstrate that our approach sets a new state-of-the-art in zero-shot video depth estimation. We offer models of different scales to support a range of scenarios, with our smallest model capable of real-time performance at 30 FPS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。