提出新型模型框架，让大模型推理时内存占用可控且不降性能。
A Method for Building Large Language Models with Predefined KV Cache Capacity
- 采用有限长度的键值缓存机制，动态更新注意力向量序列。
- 实验显示内存占用显著降低，推理质量与吞吐量保持稳定。
- 适合需要低内存高效率推理的部署场景，如边缘设备或实时服务。
本文提出一种新方法——受限缓存变压器（Bounded-Cache Transformer, BCT），用于构建具有预设键值（KV）缓存容量的大语言模型。BCT通过实现长度受限的KV缓存，解决传统方法中缓存内存消耗过大的问题，特别适用于Transformer解码仅架构中的注意力层。通过动态更新键值向量序列，BCT在有限缓存容量下实现高效推理，显著降低内存使用，同时保持模型性能与系统吞吐量。实验结果表明，该方法在维持推理质量的前提下大幅减少内存占用，为大语言模型的高效推理提供了新方案。
原文摘要 · Abstract (English)
This paper introduces a novel approach, the Bounded-Cache Transformer (BCT), for building large language models with a predefined Key-Value (KV) cache capacity. The BCT addresses the excessive memory consumption issue in traditional KV caches by implementing a bounded-length KV cache, which is particularly suitable for the attention layers in Transformer decode-only architectures. By dynamically updating the key-value vector sequences, the BCT achieves efficient inference within limited cache capacity, significantly reducing memory usage while maintaining model performance and system throughput. Experimental results demonstrate that the BCT significantly reduces memory usage while maintaining the model's inference quality, offering a new solution for efficient inference in large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。