arXiv:2609.03494cs.AI2026-09

动态分配显存,让大模型推理更省内存、更快出结果。

GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving

论文配图:GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving
图 1 · 摘自论文原文
  • 根据推理过程实时调整显存使用,不固定容量
  • 实验显示吞吐量提升最高达1.8倍
  • 适合需要长时间推理的场景,如代码生成

长输出推理使键值(KV)缓存成为高效大模型服务的关键内存瓶颈。现有压缩方法通常依赖预设的每请求显存预算,仅决定保留哪些KV状态,总容量在解码过程中固定不变。但推理任务的需求变化显著:不同请求所需显存容量不同,且单个请求的注意力需求随生成过程演变。我们提出【GrowPage】,一种按需分配KV显存的框架,将显存容量视为运行时资源。GrowPage维护轻量级双时间尺度查询摘要,捕捉近期与长期注意力行为,并通过其相对注意力工作集估算需求演化。在每个容量边界处,当检测到更大需求时,它可向系统申请额外物理页,或在当前分配内压缩KV状态。结合PagedAttention的页级内存抽象,GrowPage保持连续批处理和前缀缓存能力。在多个模型及推理基准上的实验表明,相比现有方法,GrowPage在性能-吞吐率权衡上表现更优。

原文摘要 · Abstract (English)

Long-output reasoning has made the key--value (KV) cache a critical memory bottleneck for efficient LLM serving. Existing KV compression methods usually rely on a predefined per-request budget and adjust only which KV states are retained, leaving the total capacity fixed throughout decoding. However, reasoning workloads exhibit substantial demand variation: different requests require different KV capacities, and the attention demand of an individual request evolves during generation. We introduce \textbf{GrowPage}, an on-demand KV budgeting framework that treats KV capacity as a runtime resource. GrowPage maintains lightweight dual-timescale query summaries to capture recent and long-term attention behaviors, and uses their relative attention working sets to estimate demand evolution. At each capacity boundary, GrowPage either compresses KV states within the current allocation or acquires an additional physical page when broader demand emerges. By integrating with PagedAttention's page-level memory abstraction, GrowPage preserves continuous batching and prefix caching. Experiments on reasoning benchmarks across multiple models show that GrowPage achieves a superior performance--throughput trade-off over existing approaches.

大模型推理显存优化动态调度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。