用积分噪声提升视频生成时的帧间一致性，减少闪烁和纹理粘连。
How I Warped Your Noise: a Temporally-Correlated Noise Prior for Diffusion Models
- 提出∫-噪声：将像素值视为连续噪声场的积分，而非离散采样。
- 设计定向传输方法，保持帧间噪声相关性，抑制高频闪烁。
- 适用于视频修复、条件生成等任务，适合视频生成研究者参考。
视频编辑与生成常依赖预训练的图像扩散模型。然而，扩散过程中的噪声采样方法通常忽略视频帧间的时序相关性，导致结果出现高频闪烁或纹理粘连，难以通过后处理修复。为此，本文提出一种新方法，通过新型噪声表示∫-噪声（integral noise）保留噪声序列的时序相关性。该方法将单个噪声样本重新解释为连续的无限分辨率噪声场的积分：像素值代表其所在区域对底层噪声的积分，而非离散数值。同时，设计了精细的传输机制，利用∫-噪声在帧间精确传递噪声，最大化帧间相关性并保持噪声特性。实验表明，∫-噪声可应用于视频修复、代理渲染及条件视频生成等多种任务。更多视频效果见https://warpyournoise.github.io/。
原文摘要 · Abstract (English)
Video editing and generation methods often rely on pre-trained image-based diffusion models. During the diffusion process, however, the reliance on rudimentary noise sampling techniques that do not preserve correlations present in subsequent frames of a video is detrimental to the quality of the results. This either produces high-frequency flickering, or texture-sticking artifacts that are not amenable to post-processing. With this in mind, we propose a novel method for preserving temporal correlations in a sequence of noise samples. This approach is materialized by a novel noise representation, dubbed $\int$-noise (integral noise), that reinterprets individual noise samples as a continuously integrated noise field: pixel values do not represent discrete values, but are rather the integral of an underlying infinite-resolution noise over the pixel area. Additionally, we propose a carefully tailored transport method that uses $\int$-noise to accurately advect noise samples over a sequence of frames, maximizing the correlation between different frames while also preserving the noise properties. Our results demonstrate that the proposed $\int$-noise can be used for a variety of tasks, such as video restoration, surrogate rendering, and conditional video generation. See https://warpyournoise.github.io/ for video results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。