不改模型结构，10倍加速4K视频生成
Transform Trained Transformer: Accelerating Naive 4K Video Generation Over 10$\times$
- 用多尺度共享权重窗口注意力优化推理逻辑
- 4K视频生成速度提升10倍以上，质量提升4.29分
- 适合追求高效4K视频生成的研究者和开发者
原生4K（2160×3840）视频生成面临全注意力机制随时空分辨率增加导致的二次计算爆炸问题，难以兼顾效率与质量。本文提出一种新型Transformer重构策略T3（Transform Trained Transformer），在不改变全注意力预训练模型核心架构的前提下，通过优化前向逻辑显著降低计算开销。T3-Video引入多尺度权重共享窗口注意力机制，并结合层级分块与轴保持的全注意力设计，仅需少量计算和数据即可实现预训练模型的“注意力模式”转化。在4K-VBench评测中，T3-Video不仅性能大幅提升（VQA+4.29，VTC+0.08），更将原生4K视频生成速度加速超过10倍。
原文摘要 · Abstract (English)
Native 4K (2160$\times$3840) video generation remains a critical challenge due to the quadratic computational explosion of full-attention as spatiotemporal resolution increases, making it difficult for models to strike a balance between efficiency and quality. This paper proposes a novel Transformer retrofit strategy termed $\textbf{T3}$ ($\textbf{T}$ransform $\textbf{T}$rained $\textbf{T}$ransformer) that, without altering the core architecture of full-attention pretrained models, significantly reduces compute requirements by optimizing their forward logic. Specifically, $\textbf{T3-Video}$ introduces a multi-scale weight-sharing window attention mechanism and, via hierarchical blocking together with an axis-preserving full-attention design, can effect an "attention pattern" transformation of a pretrained model using only modest compute and data. Results on 4K-VBench show that $\textbf{T3-Video}$ substantially outperforms existing approaches: while delivering performance improvements (+4.29$\uparrow$ VQA and +0.08$\uparrow$ VTC), it accelerates native 4K video generation by more than 10$\times$. Project page at https://zhangzjn.github.io/projects/T3-Video
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。