arXiv:2608.20397cs.AI2026-08

让大模型工具调用更快更省内存,支持上千工具无压力

Nexus: Depth-Adaptive KV-Cache Splicing and Retrieval-Decoupled Tool Routing for Agentic LLMs on Unified Memory

  • 用语义检索替代重复编码,工具路由不随工具增多变慢
  • 在250个工具时仍保持89%准确率,首词生成快1.66倍
  • 适合需要高效调用大量工具的智能体应用

基于模型上下文协议(MCP)的智能体大模型在每轮交互中需重新编码完整工具模板,导致预填充时间随序列长度呈平方增长,严重影响首次输出延迟(TTFT)。Nexus的核心策略是将路由与模板预填充解耦:采用INT8语义查找缓冲区(SLB)结合校准交叉编码器门限,通过检索选择工具;参数生成基于压缩文本签名(中位数19词),而非拼接键值缓存。该路径与深度无关,在工具库扩展至250个时仍维持近89%路由准确率(基线全拼接方案已溢出上下文窗口),且比完整重预填充提前1.66倍生成首个参数,主上下文节省约80%。作为次要优化,将编译后的工具模板块直接插入活跃上下文。受旋转位置编码(RoPE)相位漂移限制,锚点外插入会破坏注意力,故当阈值P=256后,Nexus通过深度自适应后缀重解码修复,最坏情况触发全量重预填充。整体保证输出一致性(顶1匹配,D_KL≈0),虽延迟可低至0.98倍,但中等深度下实现1.1–1.7倍TTFT加速,深上下文趋近平衡。两个负结果界定设计边界:非锚点位置的RoPE保真度极限,以及无参考漂移门无法预测漂移(斯皮尔曼相关系数0.193)。所有实验基于一个模型组合(Qwen2.5-14B-Instruct Q4_K_M)在Apple Silicon统一内存上进行;定性边界具泛化性,定量性能依赖具体模型对。

原文摘要 · Abstract (English)

Agentic large language models (LLMs) on the Model Context Protocol (MCP) re-encode verbose tool schemas every turn, so prefill - quadratic in sequence length - dominates time-to-first-token (TTFT) as the tool registry grows. Nexus's primary lever is to decouple routing from the schema-prefill cost: an INT8 semantic lookaside buffer (SLB) with a calibrated cross-encoder margin gate selects tools by retrieval, and arguments are generated over a compressed textual signature (median 19 tokens) rather than over spliced key/value (KV) cache. This path is depth-independent: routing accuracy stays near 89% as the registry scales to 250 tools - where a concatenate-all-schemas baseline overflows the context window entirely - and it reaches a first-argument token 1.66x sooner than a full-schema re-prefill at a ~80% main-context token saving. As a secondary, bounded lever we transplant a compiled schema KV block directly into the live context. This is fundamentally limited by rotary position embedding (RoPE) phase drift: an anchored splice is output-exact, but off-anchor placement corrupts attention, so beyond a threshold P=256 Nexus repairs the seam with a depth-adaptive suffix redecode that escalates to a full re-prefill. The resulting never-regress property is a guarantee on output fidelity (top-1 agreement, D_KL approx. 0) - not on latency, which can dip to 0.98x before converging to parity - alongside a 1.1-1.7x TTFT speedup at moderate depth that narrows to parity at deep context. Two negative results bound the design: the off-anchor RoPE fidelity boundary, and the failure of a reference-free drift gate to predict drift (Spearman rho = 0.193). All measurements are from one model tuple (Qwen2.5-14B-Instruct Q4_K_M) on Apple-silicon unified memory; the qualitative boundaries generalize, while the quantitative envelope is tuple-specific.

智能体工具调用内存优化推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。