arXiv:2602.05305cs.CVcs.AI2026-02被引 2

通过复用稳定注意力结果,提升长上下文生成效率

FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion

  • 发现块内外部注意力在扩散步骤间具有冗余性,仅块内变化大
  • 缓存外部注意力结果,使推理速度提升1.44倍,注意力耗时降60%
  • 不改变原流程,适合视频生成与长文本模型加速使用

生成分钟级视频和长文本等长序列内容对现代生成模型愈发重要。块扩散通过键值缓存与分块因果推理提升推理效率,已在扩散语言模型与视频生成中广泛应用。然而在长上下文场景下,块扩散仍因持续重复计算不断增长的键值缓存中的注意力而产生显著开销。我们发现块扩散一个未被充分探索的特性:块内跨步骤注意力存在冗余。分析表明,当前块外的注意力输出在扩散步骤间保持高度稳定,而块内注意力则显著变化。基于此,我们提出FlashBlock,一种缓存块外注意力的机制,复用稳定输出,减少注意力计算与键值缓存访问,且不修改扩散过程。此外,FlashBlock与稀疏注意力正交,可作为互补的残差复用策略,在极端稀疏化下显著提升模型准确率。在扩散语言模型与视频生成任务上的实验表明,最大支持1.44倍更高的每秒生成词数,注意力时间降低最多达1.6倍,生成质量几乎无损。

原文摘要 · Abstract (English)

Generating long-form content, such as minute-long videos and extended texts, is increasingly important for modern generative models. Block diffusion improves inference efficiency via KV caching and block-wise causal inference and has been widely adopted in diffusion language models and video generation. However, in long-context settings, block diffusion still incurs substantial overhead from repeatedly computing attention over a growing KV cache. We identify an underexplored property of block diffusion: cross-step redundancy of attention within a block. Our analysis shows that attention outputs from tokens outside the current block remain largely stable across diffusion steps, while block-internal attention varies significantly. Based on this observation, we propose FlashBlock, a cached block-external attention mechanism that reuses stable attention output, reducing attention computation and KV cache access without modifying the diffusion process. Moreover, FlashBlock is orthogonal to sparse attention and can be combined as a complementary residual reuse strategy, substantially improving model accuracy under aggressive sparsification. Experiments on diffusion language models and video generation demonstrate up to 1.44$\times$ higher token throughput and up to 1.6$\times$ reduction in attention time, with negligible impact on generation quality. Project page: https://caesarhhh.github.io/FlashBlock/.

扩散模型长文本生成注意力优化高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。