无需预标注掩码,用元数据指导扩散模型恢复损坏视频
Blind Bitstream-corrupted Video Recovery via Metadata-guided Diffusion Model

- 用运动矢量和帧类型元数据生成腐蚀区域指示
- 在扩散过程中通过交叉注意力修复大范围不规则损坏
- 适合真实场景下无标注损坏视频的恢复,尤其视频传输/存储
视频比特流损坏恢复旨在修复存储或传输中受损的内容。现有方法通常依赖预定义的损坏区域掩码,但手动标注代价高且不适用于真实场景。为此,本文提出一种新的无监督视频恢复设定,摆脱对预设掩码的依赖。该设定面临两大挑战:准确识别损坏区域,以及从大范围、不规则退化中恢复内容。我们提出元数据引导的扩散模型(M-GDM),利用视频内生元数据作为腐蚀指示信号。通过双流元数据编码器分别嵌入运动矢量与帧类型,并融合为统一表征,在每个扩散步骤与损坏隐空间特征进行交叉注意力交互。为保留完整区域,设计先验驱动的掩码预测器,结合元数据与扩散先验生成伪掩码,实现完整与恢复区域的硬掩码分离与重组。为缓解不完美掩码导致的边界伪影,引入后处理优化模块,增强完整与恢复区域间的一致性。大量实验验证了方法的有效性及其在无监督视频恢复中的优势。代码已开源:https://github.com/Shuyun-Wang/M-GDM。
原文摘要 · Abstract (English)
Bitstream-corrupted video recovery aims to restore realistic content degraded during video storage or transmission. Existing methods typically assume that predefined masks of corrupted regions are available, but manually annotating these masks is labor-intensive and impractical in real-world scenarios. To address this limitation, we introduce a new blind video recovery setting that removes the reliance on predefined masks. This setting presents two major challenges: accurately identifying corrupted regions and recovering content from extensive and irregular degradations. We propose a Metadata-Guided Diffusion Model (M-GDM) to tackle these challenges. Specifically, intrinsic video metadata are leveraged as corruption indicators through a dual-stream metadata encoder that separately embeds motion vectors and frame types before fusing them into a unified representation. This representation interacts with corrupted latent features via cross-attention at each diffusion step. To preserve intact regions, we design a prior-driven mask predictor that generates pseudo masks using both metadata and diffusion priors, enabling the separation and recombination of intact and recovered regions through hard masking. To mitigate boundary artifacts caused by imperfect masks, a post-refinement module enhances consistency between intact and recovered regions. Extensive experiments demonstrate the effectiveness of our method and its superiority in blind video recovery. Code is available at: https://github.com/Shuyun-Wang/M-GDM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。