arXiv:2604.12270cs.CV2026-04

实现高清视频实时立体修复,速度提升10倍以上。

DreamStereo: Towards Real-Time Stereo Inpainting for HD Videos

论文配图:DreamStereo: Towards Real-Time Stereo Inpainting for HD Videos
图 1 · 摘自论文原文
  • 通过梯度感知视差映射,精准生成边缘连续的遮挡区域。
  • 提出稀疏感知修复框架,减少70%冗余计算,推理提速10.7倍。
  • 可在单张A100显卡上实时处理768x1280视频,帧率25FPS。

立体视频修复旨在填充变形视频中被遮挡的区域,同时保持时间一致性,仍是一个开放难题。待修复区域通常分布在物体边界上,仅占每帧极小比例,带来双重挑战:其一,缺乏高质量立体修复数据集,限制了模型学习有效修复先验;其二,现有方法对全帧统一处理,而多数像素无需修改,导致大量冗余计算。为此,我们提出三个互相关联的组件:首先引入梯度感知视差映射(GAPW),利用反向映射与坐标函数梯度生成连续边缘和光滑遮挡区;其次提出基于视差的双投影策略(PBDP),结合GAPW生成几何一致的立体修复对及精确遮挡掩码,无需依赖真实立体视频;最后设计稀疏感知立体修复(SASI),减少超70%冗余令牌,在扩散模型推理中实现10.7倍加速,结果与全量计算相当,可在单张A100 GPU上以25 FPS实现实时处理768×1280高清视频。

原文摘要 · Abstract (English)

Stereo video inpainting, which aims to fill the occluded regions of warped videos with visually coherent content while maintaining temporal consistency, remains a challenging open problem. The regions to be filled are scattered along object boundaries and occupy only a small fraction of each frame, leading to two key challenges. First, existing approaches perform poorly on such tasks due to the scarcity of high-quality stereo inpainting datasets, which limits their ability to learn effective inpainting priors. Second, these methods apply equal processing to all regions of the frame, even though most pixels require no modification, resulting in substantial redundant computation. To address these issues, we introduce three interconnected components. We first propose Gradient-Aware Parallax Warping (GAPW), which leverages backward warping and the gradient of the coordinate mapping function to obtain continuous edges and smooth occlusion regions. Then, a Parallax-Based Dual Projection (PBDP) strategy is introduced, which incorporates GAPW to produce geometrically consistent stereo inpainting pairs and accurate occlusion masks without requiring stereo videos. Finally, we present Sparsity-Aware Stereo Inpainting (SASI), which reduces over 70% of redundant tokens, achieving a 10.7x speedup during diffusion inference and delivering results comparable to its full-computation counterpart, enabling real-time processing of HD (768 x 1280) videos at 25 FPS on a single A100 GPU.

视频修复立体生成实时处理扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。