无需训练即可生成高质量长视频,解决模型因长度不匹配导致的画质下降问题。
Free-Lunch Long Video Generation via Layer-Adaptive O.O.D Correction
- 通过分层自适应机制识别模型对时间位置和上下文长度偏差的敏感度
- 提出视频相对位置重编码与分层稀疏注意力,有效提升长视频连贯性与画质
- 适用于已有扩散模型的快速适配,适合视频生成研究者与应用开发者
使用预训练视频扩散模型生成长视频面临巨大挑战,因其通常在短片段上训练,直接用于长视频推理会导致视觉质量显著下降。本文指出问题主要源于两个分布外(O.O.D)现象:帧级相对位置O.O.D与上下文长度O.O.D。为此,提出FreeLOC——一种无需训练、分层自适应的框架,包含两项核心技术:基于视频的相对位置重编码(VRPR),通过多粒度层级重编码时间相对位置以匹配模型预训练分布;分层稀疏注意力(TSA),通过在不同时间尺度上结构化注意力密度,同时保留局部细节与长程依赖。关键在于引入分层自适应探测机制,识别各Transformer层对O.O.D问题的敏感性,实现方法的选择性高效应用。大量实验表明,该方法显著优于现有无训练方案,在时间一致性与视觉质量上达到当前最优表现。代码已开源:https://github.com/Westlake-AGI-Lab/FreeLOC。
原文摘要 · Abstract (English)
Generating long videos using pre-trained video diffusion models, which are typically trained on short clips, presents a significant challenge. Directly applying these models for long-video inference often leads to a notable degradation in visual quality. This paper identifies that this issue primarily stems from two out-of-distribution (O.O.D) problems: frame-level relative position O.O.D and context-length O.O.D. To address these challenges, we propose FreeLOC, a novel training-free, layer-adaptive framework that introduces two core techniques: Video-based Relative Position Re-encoding (VRPR) for frame-level relative position O.O.D, a multi-granularity strategy that hierarchically re-encodes temporal relative positions to align with the model's pre-trained distribution, and Tiered Sparse Attention (TSA) for context-length O.O.D, which preserves both local detail and long-range dependencies by structuring attention density across different temporal scales. Crucially, we introduce a layer-adaptive probing mechanism that identifies the sensitivity of each transformer layer to these O.O.D issues, allowing for the selective and efficient application of our methods. Extensive experiments demonstrate that our approach significantly outperforms existing training-free methods, achieving state-of-the-art results in both temporal consistency and visual quality. Code is available at https://github.com/Westlake-AGI-Lab/FreeLOC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。