arXiv:2605.27744cs.AI2026-05被引 1

为多智能体大模型服务设计动态运行时层,提升缓存效率与系统性能。

A Policy-Driven Runtime Layer for Agentic LLM Serving

  • 在框架与引擎间新增智能体运行时层,统一管理跨层策略
  • 实测缓存命中率提升13%~37%,平均首字延迟降低12%~29%
  • 适合大规模多智能体系统部署者及推理优化研究者

多智能体大模型系统已成为主流生产负载,但其服务架构并未为此类场景设计。上层智能体框架知晓智能体身份、角色与调度结构,却无法感知底层引擎事件;下层服务引擎能捕获所有事件,却对智能体一无所知。众多关键策略(如前缀缓存、批处理优化、推测执行、公平性、工具结果记忆化、安全控制等)均需同时依赖双层信息,目前只能通过临时修补解决。本文主张以架构变革替代零散修复:在框架与引擎间引入第三层——智能体运行时层,提供观察、评分、预测、行动四类原语,以智能体身份为共同坐标,使各类智能体感知策略可插拔式集成。文中将九项具体策略映射至该层,并深度验证其中影响最大的缓存策略:跨会话KV缓存优化(实现为CacheScout),通过在线学习每工作负载的智能体转移矩阵,实现基于生存性的缓存淘汰与步骤间预取。在五个真实多智能体工作负载上的初步实验表明,相比未修改的服务栈,缓存命中率提升13%~37个百分点,平均首字时间降低12%~29%,吞吐量提高6%~14%。

原文摘要 · Abstract (English)

Multi-agent LLM systems have become the dominant production workload, but the serving stack was not built for them. The agent framework above knows agent identities, role, schemas, and dispatch structure but never sees an engine-level event; the serving engine below sees every event but knows nothing about agents. A surprising number of cross-cutting policies depend on both: prefix caching, batch shaping, speculative execution, fairness, tool-result memoization, safety enforcement, and more. Each lives in the seam between the two layers and is currently solved by a one-off patch into one neighbor or the other. We argue this seam is best addressed by an architectural change rather than point fixes: insert a third tier, an agent runtime layer, between the framework and the engine, exposing four primitives (observe, score, predict, act) into which any agent-aware policy plugs, with agent identity as the shared coordinate. We map nine concrete policies onto the layer and validate the abstraction in depth on the one with the largest immediate serving-cost lever: KV caching across sessions, instantiated as CacheScout, which learns the per-workload agent transition matrix online and uses it for survival-based eviction and between-step prefetch. Preliminary results on five real multi-agent workloads show +13 to +37 pp cache hit-rate lift, 12% to 29% lower mean TTFT, and 6% to 14% higher throughput over an unmodified serving stack.

大模型服务智能体系统缓存优化运行时架构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。