通过系统实验发现，跨层共享键值缓存能显著提升大模型推理效率。
A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference
- 构建统一框架，整合多种跨层键值缓存共享方法及其变体。
- 缓存减半时，多数方法吞吐量高于标准Transformer且性能接近。
- 高层查询配低层键值更优，但需额外训练与预填充开销。
近期研究表明，跨层共享键值（KV）缓存可有效提升大语言模型（LLM）的推理效率。为系统性地研究不同跨层KV共享技术，我们提出一个统一框架，涵盖多种近期方法及其新颖变体。在该框架所有配置上进行综合实验，评估其生成吞吐量以及语言建模和下游任务的表现。结果表明，当将KV缓存大小减少2倍时，多数配置的吞吐量高于标准Transformer，同时保持有竞争力的性能。进一步压缩缓存时，将所有层的查询与上层的键值配对表现更优，但需额外训练成本和预填充延迟。本工作旨在帮助用户更明智地选择跨层KV共享策略，并推动高效LLM推理的后续研究。
原文摘要 · Abstract (English)
Recently, sharing key-value (KV) cache across layers has been found effective in efficient inference of large language models (LLMs). To systematically investigate different techniques of cross-layer KV sharing, we propose a unified framework that covers several recent methods and their novel variants. We conduct comprehensive experiments on all the configurations of the framework, evaluating their generation throughput and performance in language modeling and downstream tasks. We find that when reducing the size of the KV cache by 2$\times$, most configurations can achieve higher throughput than standard transformers while maintaining competitive performance. When further reducing the size of the KV cache, however, pairing queries of all layers with KVs of upper layers performs better, at the expense of additional training cost and prefilling latency. We hope that this work will help users make more informed choices of cross-layer KV sharing approaches and facilitate future research on efficient LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。