用图像扩散模型解决视频逆问题，避免闪烁和不一致。
Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models
- 将帧视为二维空间的连续函数，建模视频为连续形变序列。
- 在8倍超分辨率和视频修复任务中优于基于噪声变换的方法。
- 仅需图像模型即可实现时序一致生成，适合部署SOTA模型。
直接将图像扩散模型用于视频逆问题常导致闪烁、纹理粘连和时序不一致。本文将帧视为二维空间的连续函数，视频视为不同帧间的连续形变序列。基于此视角，我们仅在图像上训练函数空间扩散模型，并利用其求解时序相关的逆问题。模型需对底层空间变换保持等变性。为保证时序一致性，引入简单后处理测试时引导，以获得（自）等变解。该方法可直接部署如Stable Diffusion XL等先进潜空间扩散模型解决视频逆问题。在视频修复与8倍超分辨率任务中，性能超越基于噪声变换的现有方法。生成结果详见：https://giannisdaras.github.io/warped_diffusion.github.io/
原文摘要 · Abstract (English)
Using image models naively for solving inverse video problems often suffers from flickering, texture-sticking, and temporal inconsistency in generated videos. To tackle these problems, in this paper, we view frames as continuous functions in the 2D space, and videos as a sequence of continuous warping transformations between different frames. This perspective allows us to train function space diffusion models only on images and utilize them to solve temporally correlated inverse problems. The function space diffusion models need to be equivariant with respect to the underlying spatial transformations. To ensure temporal consistency, we introduce a simple post-hoc test-time guidance towards (self)-equivariant solutions. Our method allows us to deploy state-of-the-art latent diffusion models such as Stable Diffusion XL to solve video inverse problems. We demonstrate the effectiveness of our method for video inpainting and $8\times$ video super-resolution, outperforming existing techniques based on noise transformations. We provide generated video results: https://giannisdaras.github.io/warped_diffusion.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。