arXiv:2409.02574cs.CVcs.AI2024-09ICLR被引 14

用图像扩散模型解决视频逆问题,实现高效高质量视频修复。

Solving Video Inverse Problems Using Image Diffusion Models

  • 将视频时间维度当作图像批处理,复用现有图像扩散模型。
  • 通过批次一致性采样,提升视频帧间时序连贯性,重建效果领先。
  • 无需训练视频扩散模型,适合快速部署于各类视频修复任务。

基于扩散模型的逆问题求解器(DIS)在图像超分辨率、去模糊、图像修复等任务中已达到顶尖水平。然而,由于视频扩散模型训练困难,其在时空退化导致的视频逆问题中的应用仍不充分。为此,本文提出一种创新的视频逆问题求解方法,仅依赖图像扩散模型。受最近分解扩散采样器(DDS)启发,该方法将视频的时间维度视为图像扩散模型的批量维度,在每个反向扩散步骤中对去噪后的时空批次进行联合优化。此外,引入批次一致性扩散采样策略,通过同步图像扩散模型中的随机噪声分量,增强批次间的时序一致性。该方法在每一步反向扩散中协同优化去噪时空批次与保持批次一致性,形成新颖高效的视频逆问题扩散采样策略。实验表明,该方法能有效应对多种时空退化,实现当前最优的视频重建效果。

原文摘要 · Abstract (English)

Recently, diffusion model-based inverse problem solvers (DIS) have emerged as state-of-the-art approaches for addressing inverse problems, including image super-resolution, deblurring, inpainting, etc. However, their application to video inverse problems arising from spatio-temporal degradation remains largely unexplored due to the challenges in training video diffusion models. To address this issue, here we introduce an innovative video inverse solver that leverages only image diffusion models. Specifically, by drawing inspiration from the success of the recent decomposed diffusion sampler (DDS), our method treats the time dimension of a video as the batch dimension of image diffusion models and solves spatio-temporal optimization problems within denoised spatio-temporal batches derived from each image diffusion model. Moreover, we introduce a batch-consistent diffusion sampling strategy that encourages consistency across batches by synchronizing the stochastic noise components in image diffusion models. Our approach synergistically combines batch-consistent sampling with simultaneous optimization of denoised spatio-temporal batches at each reverse diffusion step, resulting in a novel and efficient diffusion sampling strategy for video inverse problems. Experimental results demonstrate that our method effectively addresses various spatio-temporal degradations in video inverse problems, achieving state-of-the-art reconstructions. Project page: https://svi-diffusion.github.io/

视频修复扩散模型逆问题时序一致

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。