为智能体推理设计可编程的缓存编辑机制,提升效率与灵活性。
Leyline: KV Cache Directives for Agentic Inference
- 提出Leyline指令四元组,实现对缓存内容的精准删除或替换。
- 在debug-gym上使智能体求解成功率提升14.3个百分点,延迟降低241毫秒。
- 支持策略驱动的缓存操作,适合构建复杂决策型大模型应用。
现代KV缓存管理假设聊天机器人工作负载:提示一次性到达,缓存仅追加增长,因此前缀缓存和单向淘汰是正确设计。而智能体大模型打破了这一假设,其对话通过策略驱动的编辑演化:失败的工具调用重试、过时输出丢弃、轨迹切换。这带来两个缓存问题。第一,相同内容在回合间位置变动,导致精确前缀缓存失效,尽管底层键值仍有效;已有工作如MLA的位置无关缓存解决此复用问题。第二,也是本文重点,策略需能主动指令服务系统移除或替换一段缓存内容,并继续运行而不重新预填充后续全部内容。现有机制均无法实现。生产级智能体系统只能每次修改后全量重预填充,付出完整前缀计算代价;内核级淘汰方法自行决策,无法接收外部策略指令。本文提出Leyline,一种服务端原语,填补该空白。一个声明式指令四元组将要编辑的内容与保持位置正确性的方法分离。策略声明编辑动作及其模式(就地拼接或前缀修剪式重预填充以实现语义遗忘);架构无关接口路由至各架构专用内核,通过闭式旋转位置编码(RoPE-rotation)修正恢复注意力数学一致性。拼接内核使重播命中率提升11.2个百分点,延迟最高降低241毫秒。一条十行截断规则经同一接口处理,使debug-gym上的智能体求解率提升14.3个百分点。该机制开放,其所开启的策略空间即为未来方向。
原文摘要 · Abstract (English)
Modern KV cache management assumes the chatbot workload: prompts arrive once and the cache grows append-only, so prefix caching and forward-only eviction are correct by construction. Agentic LLMs break this assumption. Their conversations evolve through policy-driven editing: failed tool calls are retried, stale outputs dropped, trajectories pivoted. Two distinct cache problems result. First, identical content moves to new positions between turns, invalidating exact-prefix caches even though the underlying KV would still be valid; recent work on position-independent caching for MLA addresses this reuse problem. Second, and this paper's focus, a policy may need to direct the serving system to actively remove or replace a span of cached content and continue without re-prefilling everything that came after. No existing primitive offers this. Production agentic harnesses fall back to re-prefill on every edit, paying full prefix-recomputation cost; kernel-level eviction methods make their own decisions and cannot accept policy directives from outside the kernel. We introduce Leyline, a serving-side primitive that closes this gap. A declarative directive 4-tuple separates what to edit from how to preserve position correctness. The policy declares the edit and its mode (in-place splice or prefix-trimmed re-prefill for semantic forgetting); an architecture-agnostic interface routes to a per-architecture kernel that restores attention math via a closed-form RoPE-rotation correction. The splice kernel lifts replay cache-hit by +11.2 pp and cuts latency by up to 241 ms. A ten-line truncation rule routed through the same interface lifts agentic solve rate by +14.3 pp on debug-gym. The mechanism is open; the policy space it enables is the agenda.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。