动态冻结低重要性词元的键值缓存，实现大模型推理内存近线性下降。
Adaptive Soft Rolling KV Freeze with Entropy-Guided Recovery: Sublinear Memory Growth for Efficient LLM Inference
- 滑动窗口内识别低重要性词元并软冻结其键值更新
- 主动缓存减少55%-67%的活跃键值缓存，保持生成质量
- 无需微调，适用于任意架构，适合长上下文部署
我们提出一种无需训练的推理时高效大语言模型生成框架ASR-KF-EGR。该方法引入可逆软冻结机制，在滑动注意力窗口内识别低重要性词元，并临时暂停其键值（KV）更新。与永久丢弃上下文的淘汰式方法不同，ASR-KF-EGR将所有词元保留在离显存存储中，按需恢复。我们进一步设计次线性冻结调度策略，使冻结时长随低重要性检测次数次线性增长，避免过度压缩。在LLaMA-3 8B上的初步实验表明，该方法可实现55%-67%的活跃键值缓存缩减，同时维持生成质量并通过针入 haystack 检索测试。该方法架构无关、无需微调，为长上下文大模型的内存受限部署提供了实用解决方案。
原文摘要 · Abstract (English)
We present Adaptive Soft Rolling KV Freeze with Entropy-Guided Recovery (ASR-KF-EGR), a training-free inference-time framework for efficient large language model generation. Our method introduces a reversible soft-freeze mechanism that temporarily suspends key-value (KV) updates for low-importance tokens identified within a sliding attention window. Unlike eviction-based approaches that permanently discard context, ASR-KF-EGR preserves all tokens in off-GPU storage and restores them on demand. We extend the framework with sublinear freeze scheduling, where freeze duration grows sublinearly with repeated low-importance detections, preventing over-aggressive compression. Preliminary experiments on LLaMA-3 8B demonstrate 55-67% reduction in active KV cache size while maintaining generation quality and passing needle-in-haystack retrieval tests. The method is architecture-agnostic, requires no fine-tuning, and provides a practical solution for memory-constrained deployment of long-context LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。