通过动态终止冗余注意力计算，提升大模型长文本生成效率
ART: Attention Run-time Termination for Efficient Large Language Model Decoding
- 运行时监测注意力输出，及时停止无效的键值块访问
- 在不降低质量的前提下，生成吞吐量最高提升20%
- 适合追求推理效率的长文本生成场景
大语言模型在长上下文生成中受限于键值（KV）缓存的访问与处理成本。尽管注意力输出同时依赖键和值，但现有大多数KV管理方法仅基于键进行剪枝，因引入值会带来高昂开销。本文提出注意力运行时终止（ART），一种轻量级运行时机制，在内核执行过程中追踪累积注意力输出，一旦后续贡献可忽略即终止后续KV块访问。ART不替代现有密集或稀疏注意力策略，而是在其基础上动态终止冗余的KV遍历。我们提出基于稳定性的判定准则，监控中间注意力输出的幅值与方向变化，并对截断误差提供理论分析。在LongBench和RULER Needle-in-a-Haystack任务上的实验表明，ART使现有KV缓存方法的生成吞吐量提升高达20%，且不影响生成质量。
原文摘要 · Abstract (English)
Long-context decoding in Large Language Models (LLMs) is constrained by the cost of accessing and processing the Key-Value (KV) cache. Despite evidence that attention outputs depend jointly on keys and values, most existing KV management methods rely on key-only pruning, since incorporating values incurs prohibitive overhead. In this paper, we propose Attention Run-time Termination (ART), a lightweight run-time mechanism that tracks accumulated attention outputs during kernel execution and terminates subsequent KV block accesses once further contributions become negligible. Rather than replacing KV selection, ART dynamically terminates redundant KV traversal on top of existing dense or sparse attention policies. We introduce a stability-based criterion that monitors both magnitude and directional changes of intermediate attention outputs and provideds a theoretical characterization of the resulting truncation error. Experiments on the LongBench and RULER Needle-in-a-Haystack tasks show that ART increases the generation throughput of existing KV-cache methods by up to 20%, without compromising the result quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。