发现视频生成模型中注意力的内在稀疏结构，实现零训练高效加速。
ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers
- 基于权重编码提取稳定的稀疏注意力拓扑，离线生成无需运行时搜索。
- 在Wan2.1上实现1.90倍端到端加速，保持高画质生成质量。
- 适合追求高效视频生成的开发者与部署场景，尤其适合硬件优化。
尽管扩散变压器（DiTs）推动了高质量视频生成的发展，但其依赖3D全注意力机制导致计算复杂度呈二次增长。现有稀疏化方法面临两难：动态剪枝带来高昂运行开销和内存碎片，静态启发式规则无法捕捉细粒度依赖关系。本文提出ScalingAttention，一种无训练框架，基于关键归纳偏置——虽然单个激活值依赖输入，但每个注意力头的高密度注意力区域会迅速收敛至稳定、提示无关的内在稀疏拓扑。该拓扑由权重编码，具有尺度不变性且易于提取。ScalingAttention通过WEST（权重编码稀疏拓扑）离线提取鲁棒的块稀疏先验掩码，消除运行时搜索；通过FAST（保真度感知敏感度调优）自适应调整头级稀疏度以满足扩散保真度需求。为确保实际加速效果，我们协同设计了硬件对齐的位级块稀疏内核。在Wan2.1数据集上的实验表明，相比当前最优基线，可实现最高1.90倍端到端加速，同时保持更优保真度，确立了新的帕累托前沿。
原文摘要 · Abstract (English)
While Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, their reliance on 3D full attention creates a quadratic computational bottleneck. Existing sparse methods face a dilemma: dynamic pruning suffers from prohibitive runtime overhead and memory fragmentation, while static heuristics fail to capture fine-grained dependencies. In this work, we propose ScalingAttention, a training-free framework grounded in a key inductive bias: while individual activations are input-dependent, the high-mass attention regions for each head rapidly converge to a stable, prompt-agnostic Intrinsic Sparse Topology. This topology is weight-encoded, scale-invariant, and efficient to extract. ScalingAttention decouples topology discovery from sparsity control via: (1) WEST (Weight-Encoded Sparse Topology), which extracts a robust block-sparse prior mask offline to eliminate runtime search; (2) FAST (Fidelity-Aware Sensitivity Tuning), which adaptively tunes head-wise sparsity based on diffusion fidelity requirements. To ensure practical acceleration, we co-design a hardware-aligned bit-wise block-sparse kernel. Experiments on Wan2.1 show up to 1.90X end-to-end speedup with superior fidelity, establishing a new Pareto frontier over state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。