arXiv:2608.28276cs.LG2026-08

通过解析器结构信息优化关键词值缓存,提升大模型生成结构化数据的准确率。

Parser States Already Know: Structure-Conditioned KV Persistence for Structured Generation

论文配图:Parser States Already Know: Structure-Conditioned KV Persistence for Structured Generation
图 1 · 摘自论文原文
  • 利用解析器状态决定每层组的键值缓存保留策略。
  • 在0.33总缓存预算下,平均比基线高17.39个百分点。
  • 适合需要高可靠结构化输出的场景,如函数调用与SQL生成。

结构化生成是大型语言模型代理生成JSON、SQL和函数调用的基础,一个字段错误就可能导致下游操作失败。约束解码已追踪解析器转移以保证格式有效性,这些转移揭示了生成标记如何参与关键决策,如必填字段、参数和结构边界。现有键值(KV)压缩大多忽略这一任务相关结构信号。本文提出PASK(Parser-Aware Structural KV Persistence),将解析器衍生的结构信息转化为分层组特定的KV持久化决策。PASK通过任务错误敏感性设定最低保护阈值,利用注意力输出失真分配剩余的KV容量,在离线校准阶段构建持久化策略,线上仅需轻量级结构条件查找。在总KV预算为0.33时,PASK在Qwen3-4B上的八个BFCL非实时和实时子类别上平均超越最强压缩基线17.39个百分点。端到端服务中,吞吐量最高提升2.2倍,端到端延迟降低3.3倍,峰值GPU内存仅需全量缓存的0.53倍。

原文摘要 · Abstract (English)

Structured generation underpins large language model (LLM) agents that produce JSON, SQL, and function calls, where a single wrong field can cause the downstream action to fail. Constrained decoding already tracks parser transitions to enforce formal validity, and these transitions expose how generated tokens participate in schema-critical decisions such as required fields, arguments, and structural boundaries under the active grammar. Existing KV compression largely leaves this task-relevant structural signal unused. We introduce PASK (Parser-Aware Structural KV Persistence), which turns parser-derived structure into layer-group-specific KV persistence decisions. PASK addresses the mismatch between model-side KV sensitivity and task-level structured risk by using task-error sensitivity to set minimum protection floors and attention-output distortion to allocate residual KV capacity. An offline calibration stage compiles these signals into a persistence policy, leaving only lightweight structure-conditioned lookup online. At a targe total KV budget of 0.33, PASK outperforms the strongest compressed baseline by 17.39 percentage points on average across eight BFCL non-live and Live subcategories on Qwen3-4B. In end-to-end serving, PASK achieves up to 2.2x higher throughput and 3.3x lower TPOT, while using 0.53x the peak GPU memory of Full KV.

结构化生成键值缓存大模型优化解析器感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。