arXiv:2605.05696cs.DCcs.AI2026-05被引 2

解决智能体大模型服务中缓存失效问题,提升推理效率。

Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving

论文配图:Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving
图 1 · 摘自论文原文
  • 用内容哈希和旋转规则实现无位置依赖的缓存机制
  • 在三种模型上恢复超80%提示令牌,预填充能耗降低63%
  • 适合大规模智能体应用的高效推理部署

智能体类大模型工作负载每轮将比特相同的标记移位,导致前缀缓存从第一个差异字节起失效。运营商报告缓存命中率下降,引发从轻微延迟到高达10-16秒的首字延迟飙升,且内容未变。现有无位置依赖缓存系统需对完整$d_K$维键进行RoPE修正,这是GQA架构带来的开销,非缓存本身所需。多头潜在注意力(MLA)结构在DeepSeek-V2/V3/R1、Kimi-K2/Moonlight、GLM-5及Mistral Large 3中已规模化部署,将每个KV行分解为与位置无关的$c_{KV}$和可闭式修正的64维$k_r$,使基于内容的缓存成为自然选择而非GQA绕行方案。我们提出Irminsul,扩展SGLang的基数缓存,采用CDC分段内容哈希键和针对$k_r$的$δ$-旋转规则。评估三个原生MLA-MoE部署——DeepSeek-V2-Lite(16B/2.4B)、Kimi Moonlight-16B-A3B和JoyAI-Flash(48B/3B)——在全部三者上保持输出一致性,两端测量恢复效果;Irminsul在智能体流量下可恢复约83%的提示令牌(高于精确前缀),每缓存命中节省63%预填充能耗。我们认为内容地址缓存应作为推理栈中的原生基础组件,而非对前缀匹配的补丁。

原文摘要 · Abstract (English)

Agentic LLM workloads put bit-identical tokens at shifted positions every turn, voiding prefix caches at the first byte of divergence. Operators report cache-hit regressions ranging from moderate slowdowns to severe TTFT spikes of 10-16s on unchanged content. Prior position-independent caching systems correct RoPE on the full $d_K$-dimensional key, an architectural cost imposed by GQA, not by caching itself. Multi-Head Latent Attention, deployed at scale in DeepSeek-V2/V3/R1, Kimi-K2/Moonlight, GLM-5, and Mistral Large 3, factors each KV row into a position-free $c_{KV}$ and a 64-dim $k_r$ correctable in closed form; this structure motivates content-addressed caching as a natural fit rather than a GQA workaround. We present Irminsul, which extends SGLang's radix cache with content-hash keying over CDC-chunked segments and a $δ$-rotation rule for $k_r$. We evaluate three native MLA-MoE deployments - DeepSeek-V2-Lite (16B/2.4B), Kimi Moonlight-16B-A3B, and JoyAI-Flash (48B/3B) - with output-consistency on all three and recovery measured on the two endpoints; Irminsul recovers up to ~83% of prompt tokens above exact-prefix on agentic traffic while delivering 63% prefill energy savings per cache hit. We argue that content-addressed caching belongs in the serving stack as a first-class primitive, not a retrofit over prefix matching.

大模型推理缓存优化智能体系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。