通过识别对话轮次相似性，动态压缩缓存提升大模型推理效率
Round Attention: A Novel Round-Level Attention Mechanism to Accelerate LLM Inference
- 按对话轮次分组，仅保留最相关轮次的缓存数据
- 内存占用降低54%至82%，答案准确率不变
- 适合长对话场景的高效推理部署
大语言模型上下文窗口增大虽提升了处理长文本任务的能力，但随着对话轮次增多，需存储大量键值缓存（KV cache），显著影响模型服务系统的效率与可用性。本文分析真实用户对话数据，在轮次粒度上发现模型推理存在一个转折层，之后各轮次注意力分布呈现显著相似性。基于此，提出Round Attention——一种新型轮次级注意力机制，仅选择top-k相关轮次的KV缓存进行处理，k值由转折层的注意力矩阵动态确定。理论分析表明，该方法可使内存使用减少54%至82%；实验结果证实，加载稀疏的关键轮次缓存仍能保持答案准确性，无性能下降。
原文摘要 · Abstract (English)
The increasing context window size in large language models (LLMs) has improved their ability to handle complex, long-text tasks. However, as the conversation rounds continue, it is required to store a large amount of KV cache in GPU memory, which significantly affects the efficiency and even availability of the model serving systems. This paper analyzes dialogue data from real users on the granularity of round and discovers that the LLM inference manifests a watershed layer, after which the distribution of round-level attention shows notable similarity. Based on this, we propose Round Attention - a novel round-level attention mechanism that selectively processes the KV cache of top-k relevant rounds, where k is dynamically determined through the attention matrix in the watershed layer. Theoretical analysis demonstrates that our method reduces memory usage by 54\% to 82\%, while experimental results confirm that loading sparse critical-round KV cache maintains answer accuracy without performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。