无需训练即可生成超高清视频，保持细节与全局一致性。
FreeSwim: Revisiting Sliding-Window Attention Mechanisms for Training-Free Ultra-High-Resolution Video Generation
- 采用内向滑动窗口注意力机制，保留训练时的感知范围。
- 双路径结构结合全视野交叉注意力，避免重复与失真。
- 支持高效生成，性能超越部分需训练的方法。
现代基于Transformer的视频生成模型中，注意力机制的二次方时间与内存复杂度导致超高清视频端到端训练成本过高。为此，我们提出一种无需训练的方法，利用在原始分辨率下预训练的视频扩散Transformer，直接生成更高分辨率视频。核心是内向滑动窗口注意力机制，关键发现是保持查询令牌的训练尺度感受野对视觉保真度至关重要。然而，简单的局部窗口注意力常导致内容重复且缺乏全局连贯性。为此，我们设计了双路径架构，通过新型交叉注意力覆盖策略，使局部注意力产生的语义内容受全视野分支引导，确保整体一致性。此外，为提升效率，该分支引入交叉注意力缓存策略，避免频繁计算完整3D注意力。大量实验表明，本方法在无需训练的前提下实现超高清视频生成，具备精细视觉细节与高效率；在VBench上表现优于甚至媲美部分训练型方法，同时保持竞争力或更优效率。代码已公开于：https://github.com/WillWu111/FreeSwim
原文摘要 · Abstract (English)
The quadratic time and memory complexity of the attention mechanism in modern Transformer based video generators makes end-to-end training for ultra high resolution videos prohibitively expensive. Motivated by this limitation, we introduce a training-free approach that leverages video Diffusion Transformers pretrained at their native scale to synthesize higher resolution videos without any additional training or adaptation. At the core of our method lies an inward sliding window attention mechanism, which originates from a key observation: maintaining each query token's training scale receptive field is crucial for preserving visual fidelity and detail. However, naive local window attention, unfortunately, often leads to repetitive content and exhibits a lack of global coherence in the generated results. To overcome this challenge, we devise a dual-path pipeline that backs up window attention with a novel cross-attention override strategy, enabling the semantic content produced by local attention to be guided by another branch with a full receptive field and, therefore, ensuring holistic consistency. Furthermore, to improve efficiency, we incorporate a cross-attention caching strategy for this branch to avoid the frequent computation of full 3D attention. Extensive experiments demonstrate that our method delivers ultra-high-resolution videos with fine-grained visual details and high efficiency in a training-free paradigm. Meanwhile, it achieves superior performance on VBench, even compared to training-based alternatives, with competitive or improved efficiency. Codes are available at: https://github.com/WillWu111/FreeSwim
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。