提出动态立体匹配新方法,提升视频深度估计一致性与效率。
PPMStereo: Pick-and-Play Memory Construction for Consistent Dynamic Stereo Matching
- 分两阶段构建记忆模块:先筛选关键帧,再自适应加权聚合。
- 在Sintel数据集上达成0.62/1.11的TEPE,优于现有方法9%以上。
- 适合需要高一致性的实时立体视觉应用,如AR、VR场景。
从立体视频中实现时间上一致的深度估计对增强现实等真实场景应用至关重要,不一致的深度会破坏用户沉浸感。尽管重要,该任务仍具挑战性,因难以高效建模长时序一致性。现有方法通过聚合时空信息缓解问题,但存在根本权衡:有限时序建模效果有限,捕捉长程依赖则显著增加计算开销。为此,本文提出一种用于动态立体匹配的记忆缓冲机制,名为PPMStereo。受人类双阶段决策过程启发,设计了‘挑选-播放’(Pick-and-Play Memory, PPM)模块:先挑选最相关帧,再自适应加权这些帧进行时空聚合。该协同机制在保持紧凑内存的同时实现高度信息丰富的时序一致性聚合。大量实验验证其有效性,在准确性和时间一致性上均达到当前最优。特别地,于Sintel clean/final数据集上分别取得0.62/1.11的TEPE,相比BiDAStereo提升17.3%和9.02%,且计算成本更低。代码已开源。
原文摘要 · Abstract (English)
Temporally consistent depth estimation from stereo video is critical for real-world applications such as augmented reality, where inconsistent depth estimation disrupts the immersion of users. Despite its importance, this task remains challenging due to the difficulty in modeling long-term temporal consistency in a computationally efficient manner. Previous methods attempt to address this by aggregating spatio-temporal information but face a fundamental trade-off: limited temporal modeling provides only modest gains, whereas capturing long-range dependencies significantly increases computational cost. To address this limitation, we introduce a memory buffer for modeling long-range spatio-temporal consistency while achieving efficient dynamic stereo matching. Inspired by the two-stage decision-making process in humans, we propose a \textbf{P}ick-and-\textbf{P}lay \textbf{M}emory (PPM) construction module for dynamic \textbf{Stereo} matching, dubbed as \textbf{PPMStereo}. PPM consists of a `pick' process that identifies the most relevant frames and a `play' process that weights the selected frames adaptively for spatio-temporal aggregation. This two-stage collaborative process maintains a compact yet highly informative memory buffer while achieving temporally consistent information aggregation. Extensive experiments validate the effectiveness of PPMStereo, demonstrating state-of-the-art performance in both accuracy and temporal consistency. % Notably, PPMStereo achieves 0.62/1.11 TEPE on the Sintel clean/final (17.3\% \& 9.02\% improvements over BiDAStereo) with fewer computational costs. Codes are available at \textcolor{blue}{https://github.com/cocowy1/PPMStereo}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。