用PCA解耦视频长短帧信息,实现无训练长视频生成的高一致性与高质量。
FreePCA: Integrating Consistency Information across Long-short Frames in Training-free Long Video Generation via Principal Component Analysis
- 通过PCA在主成分空间中分离出一致外观与运动强度信息。
- 在不训练模型前提下,显著提升长视频生成的视觉质量与运动连贯性。
- 适用于多种扩散模型,适合追求高效长视频生成的研究者。
长视频生成需在短视频训练模型基础上生成时长更长的视频,但因帧数差异导致分布偏移。现有无训练方法难以兼顾局部短帧细节与全局长帧一致性,因视频外观与运动紧密耦合,常引发运动不一致和画质下降。本文揭示:通过主成分分析(PCA)可精确解耦全局一致外观与运动强度信息,实现二者互补融合。据此提出FreePCA——一种基于PCA的无训练长视频生成范式,通过主成分空间中的余弦相似度度量解耦特征,并渐进式融合以保持原始质量与平滑过渡;同时重用初始噪声均值统计量进一步增强一致性。实验表明,FreePCA可适配多种视频扩散模型且无需训练,带来显著性能提升。代码已开源。
原文摘要 · Abstract (English)
Long video generation involves generating extended videos using models trained on short videos, suffering from distribution shifts due to varying frame counts. It necessitates the use of local information from the original short frames to enhance visual and motion quality, and global information from the entire long frames to ensure appearance consistency. Existing training-free methods struggle to effectively integrate the benefits of both, as appearance and motion in videos are closely coupled, leading to motion inconsistency and visual quality. In this paper, we reveal that global and local information can be precisely decoupled into consistent appearance and motion intensity information by applying Principal Component Analysis (PCA), allowing for refined complementary integration of global consistency and local quality. With this insight, we propose FreePCA, a training-free long video generation paradigm based on PCA that simultaneously achieves high consistency and quality. Concretely, we decouple consistent appearance and motion intensity features by measuring cosine similarity in the principal component space. Critically, we progressively integrate these features to preserve original quality and ensure smooth transitions, while further enhancing consistency by reusing the mean statistics of the initial noise. Experiments demonstrate that FreePCA can be applied to various video diffusion models without requiring training, leading to substantial improvements. Code is available at https://github.com/JosephTiTan/FreePCA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。