解决动态稀疏注意力的缓存效率问题，提升大模型推理速度。
Dynamic Sparse Attention: Access Patterns and Architecture
- 通过记录每层注意力选择的键值索引，发现缓存碎片化严重。
- 提出层级缓存预留机制，减少缓存未命中率，提升解码吞吐。
- 适合关注大模型推理优化和系统架构设计的研究者。
动态稀疏注意力（DSA）通过仅计算缓存键值对中前k个最相关项来降低每个标记的注意力带宽，但其依赖标记的选择模式带来了系统级挑战：键值工作集碎片化、易变且难以预取，导致缓存局部性差，解码吞吐下降。我们通过在多个开源模型上实现轻量级索引器，并在自回归解码过程中记录各层键值索引，分析了这一问题。结果表明，现有DSA后端存在大量阻塞式最后一级（LL）缓存未命中事件，造成效率损失；为此我们提出一种新型LL缓存预留系统，将键值项在解码步骤间保留在LL缓存中，并结合标记粒度的LRU淘汰策略。基于收集的数据，我们验证了该架构在不同骨干网络上部署DSA时的性能收益。最后，我们提出了未来在架构与算法层面改进DSA推理服务方向的建议。
原文摘要 · Abstract (English)
Dynamic sparse attention (DSA) reduces the per-token attention bandwidth by restricting computation to a top-k subset of cached key-value (KV) entries, but its token-dependent selection pattern introduces a system-level challenge: the KV working set is fragmented, volatile, and difficult to prefetch, which can translate into poor cache locality and stalled decode throughput. We study these effects by implementing a lightweight indexer for DSA-style selection on multiple open-source backbones and logging per-layer KV indices during autoregressive decoding. Our analysis shows a gap in serving DSA backbones - a potential for a high volume of blocking LL (last level) cache miss events, causing inefficiency; we propose a novel LL cache reservation system to save KV tokens in the LL cache between decode steps, combined with a token-granularity LRU eviction policy, and show on the data we collected how this architecture can benefit serving with DSA implemented on different backbones. Finally, we propose directions for future architectural and algorithmic exploration to improve serving of DSA on modern inference platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。