无需训练即可加速视频生成，提升稀疏注意力精度与效率
Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models
- 通过响应耦合分块与探针校准重建残差，实现可执行的稀疏注意力
- 在4个模型上降低注意力重构误差，实际执行对数密度达22%-26%
- 适合追求推理加速且不希望损失生成质量的视频生成研究者
无训练的块稀疏注意力可加速视频变换器，但行级注意力集中本身无法定义可执行的稀疏算子。共享块路由的查询可能具有低重叠支持，而保留注意力质量并不能决定跳过的交互带来的后softmax误差。我们发现分块几何同时影响聚合支持和稀疏输出残差的可预测性。提出SparsePR，结合响应耦合分块与探针拟合残差重建。采样查询的键响应形成配对的K/V组，其质心诱导出共享路由的查询-响应坐标。少量精确查询行在探针残差观察到的输出子空间内，对稀疏输出进行特定调用的仿射修正。在四个异构视频生成与世界模型中，SparsePR一致降低注意力重构误差。消融实验表明，探针拟合贡献了主要改进，响应耦合分块降低硬丢弃误差，并在有限探针预算下提升重构性能。SparsePR在保持生成质量的前提下，实现22.0%-26.0%的实际执行对数密度，获得1.48x-2.61x端到端加速。
原文摘要 · Abstract (English)
Training-free block-sparse attention can accelerate video transformers, but row-wise attention concentration does not by itself specify an executable sparse operator. Queries sharing a block route may have poorly overlapping supports, while retained attention mass alone does not determine the post-softmax error from skipped interactions. We show that partition geometry affects both pooled support and the predictability of the remaining residual from the sparse output. We introduce SparsePR, which combines Response-Coupled Partitioning with Probe-Fitted Residual Reconstruction. Sampled-query key responses form paired K/V groups, whose centroids induce query-response coordinates for shared routing. A small set of exact query rows then calibrates a call-specific affine correction from the sparse output within the output subspace observed in the probe residuals. Across four heterogeneous video generation and world models, SparsePR consistently reduces attention-reconstruction error. Ablations show that probe fitting accounts for most of this reduction, while response-coupled partitioning lowers hard-drop error and improves reconstruction under a finite probe budget. SparsePR preserves generation quality at 22.0-26.0% realized executed-pair density while achieving 1.48x-2.61x end-to-end speedups. Project page: https://pardistaghavi.github.io/SparsePR-website/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。