通过压缩输入阶段的KV缓存，实现更大批次与更高推理吞吐。
Batch-Max: Higher LLM Throughput using Larger Batch Sizes and KV Cache Compression
- 在输入处理阶段压缩KV缓存，释放内存
- 支持更大批量，吞吐量显著提升
- 适合资源受限场景下的大批次推理
一些工作已开发出逐出策略，用于在推理过程中移除键值（KV）对以提高效率。现有研究主要集中在输入提示处理完成后压缩KV缓存，以加速后续标记生成。但在GPU内存有限且输入上下文长度超过生成长度的场景下，我们发现：若在输入处理阶段也压缩KV缓存，即可使用更大的批量大小，从而显著提升吞吐量，同时保持原始模型的准确性。
原文摘要 · Abstract (English)
Several works have developed eviction policies to remove key-value (KV) pairs from the KV cache for more efficient inference. The focus has been on compressing the KV cache after the input prompt has been processed for faster token generation. In settings with limited GPU memory, and when the input context is longer than the generation length, we show that by also compressing the KV cache during the input processing phase, larger batch sizes can be used resulting in significantly higher throughput while still maintaining the original model's accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。