解决图像生成视频时运动不足问题,提升动态效果而不损失画质。
Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidance
- 通过自适应低通滤波,在去噪早期调节输入图像频率,引导更自然的运动生成。
- 在VBench上平均动态度提升33%,同时保持或改善图像质量和文本对齐。
- 无需训练,可直接部署于现有模型,适合需要流畅视频生成的场景。
近期文本到视频(T2V)模型在生成高质量动态视频方面表现强劲。为提升视觉可控性,已有研究尝试微调预训练T2V模型以支持图像到视频(I2V)生成。然而,此类适配常抑制生成视频的运动动态,使其比T2V结果更静态。本文分析该现象,发现根源在于输入图像中高频细节过早暴露,导致采样过程偏向捷径路径,过度拟合参考图像的静态外观。为此,提出自适应低通引导(ALG),一种无需训练的I2V采样优化方法,可在不牺牲单帧图像质量的前提下增强视频动态性。具体而言,ALG在去噪早期对条件图像应用自适应低通滤波,动态调控其频率成分。大量实验表明,ALG显著提升生成视频的时间动态性,同时保持或改进图像保真度与文本对齐。例如,在VBench测试集上,各模型平均动态度提升33%,且维持原有视频质量。
原文摘要 · Abstract (English)
Recent text-to-video (T2V) models have demonstrated strong capabilities in producing high-quality, dynamic videos. To improve the visual controllability, recent works have considered fine-tuning pre-trained T2V models to support image-to-video (I2V) generation. However, such adaptation frequently suppresses motion dynamics of generated outputs, resulting in more static videos compared to their T2V counterparts. In this work, we analyze this phenomenon and identify that it stems from the premature exposure to high-frequency details in the input image, which biases the sampling process toward a shortcut trajectory that overfits to the static appearance of the reference image. To address this, we propose adaptive low-pass guidance (ALG), a simple training-free fix to the I2V model sampling procedure to generate more dynamic videos without compromising per-frame image quality. Specifically, ALG adaptively modulates the frequency content of the conditioning image by applying a low-pass filter at the early stage of denoising. Extensive experiments show ALG significantly improves the temporal dynamics of generated videos, while preserving or even improving image fidelity and text alignment. For instance, on the VBench test suite, ALG achieves a 33% average improvement across models in dynamic degree while maintaining the original video quality. For additional visualizations and source code, see the project page.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。