让单图深度估计模型在视频中更稳定,减少闪烁抖动。
StableDPT: Temporal Stable Monocular Video Depth Estimation
- 用跨帧注意力融合关键帧信息,提升时序一致性。
- 在多个数据集上实现领先精度,处理速度提升2倍。
- 适合需要实时稳定深度预测的视频应用。
将单图像单目深度估计(MDE)模型应用于视频序列时,常出现显著的时间不稳定性与闪烁伪影。本文提出StableDPT,通过引入可训练的时序模块,将任意先进图像级深度模型适配为视频处理模型,仅需单张GPU训练数日。该架构基于现成的视觉变换器(ViT)编码器,并增强密集预测变换器(DPT)解码头。核心创新在于解码头中的时序层,采用高效的跨注意力机制,整合整个视频序列中采样的关键帧信息,从而捕捉全局上下文与帧间关系,实现更准确且时间稳定的深度预测。此外,提出一种新颖的推理策略,可处理任意长度视频,避免传统重叠窗口带来的尺度错位与冗余计算。在多个基准数据集上的评估表明,该方法在时间一致性方面表现优异,达到当前最优性能,并在真实场景中实现超过2倍的处理速度提升。
原文摘要 · Abstract (English)
Applying single image Monocular Depth Estimation (MDE) models to video sequences introduces significant temporal instability and flickering artifacts. We propose a novel approach that adapts any state-of-the-art image-based (depth) estimation model for video processing by integrating a new temporal module - trainable on a single GPU in a few days. Our architecture StableDPT builds upon an off-the-shelf Vision Transformer (ViT) encoder and enhances the Dense Prediction Transformer (DPT) head. The core of our contribution lies in the temporal layers within the head, which use an efficient cross-attention mechanism to integrate information from keyframes sampled across the entire video sequence. This allows the model to capture global context and inter-frame relationships leading to more accurate and temporally stable depth predictions. Furthermore, we propose a novel inference strategy for processing videos of arbitrary length avoiding the scale misalignment and redundant computations associated with overlapping windows used in other methods. Evaluations on multiple benchmark datasets demonstrate improved temporal consistency, competitive state-of-the-art performance and on top 2x faster processing in real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。