通过分层稀疏注意力与异步缓存调度，实现长文本生成的高效高精度推理。
AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention
- 分两级稀疏策略：先块后词，兼顾效率与精度
- 在48k-96k上下文上提速1.2至10倍，吞吐提升1.3至4.7倍
- 适合需要长文本生成的部署场景，如对话系统、内容创作
大语言模型在长上下文推理中面临二次方注意力复杂度和高昂的键值缓存内存开销。虽然基于标记的稀疏注意力精度更高，但索引开销大；而基于块的方法虽更高效，却牺牲了精度。我们提出AsyncTLS，一种分层稀疏注意力系统，结合粗粒度块过滤与细粒度标记选择，在保持接近全注意力精度的同时，显著提升效率，并配备异步卸载引擎，利用时间局部性将键值缓存传输与计算重叠。在Qwen3和GLM-4.7-Flash的GQA与MLA架构上评估，于48k–96k上下文长度下，实现与全注意力相当的精度，操作速度提升1.2x–10.0x，端到端吞吐提升1.3x–4.7x。
原文摘要 · Abstract (English)
Long-context inference in LLMs faces the dual challenges of quadratic attention complexity and prohibitive KV cache memory. While token-level sparse attention offers superior accuracy, its indexing overhead is costly; block-level methods improve efficiency but sacrifice precision. We propose AsyncTLS, a hierarchical sparse attention system that combines coarse-grained block filtering with fine-grained token selection to balance accuracy and efficiency, coupled with an asynchronous offloading engine that overlaps KV cache transfers with computation via temporal locality exploitation. Evaluated on Qwen3 and GLM-4.7-Flash across GQA, and MLA architectures, AsyncTLS achieves accuracy comparable to full attention while delivering 1.2x - 10.0x operator speedups and 1.3x - 4.7x end-to-end throughput improvements on 48k - 96k contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。