解决大模型代理在云端调用工具时的扩展难题
Scalable LLM Agent Tool Access in the Cloud
- 构建网关系统分离工具调用与数据平面，统一管理兼容性
- 支持3000+工具接入，工具选择效率提升8.9倍，节省23.8倍令牌开销
- 适用于需要大规模工具调用的云原生AI代理系统
大型语言模型（LLM）代理日益依赖工具调用以操作外部系统，而模型上下文协议（MCP）已成为其事实标准接口。然而，在云规模下运行MCP面临挑战：工具提供方的旧服务无法直接通过MCP调用，快速协议迭代带来持续兼容成本；代理端受限于上下文窗口和推理开销，大量工具接入会增加令牌使用和延迟，降低任务成功率。此外，对于多副本状态化MCP后端，保持会话亲和性也增加了客户端复杂度。本文提出一种面向MCP服务的云规模网关系统，打破数据平面的直连模式，将遗留服务集成、不兼容MCP版本合并、访问控制、工具推荐和会话感知路由集中于网关。混合检索实现98%的Top-15召回率；系统可支持超过3000个工具，工具选择时间减少8.9倍，令牌消耗降低23.8倍，且每请求开销低、具备良好的横向扩展能力。最后，分享了该系统在生产环境部署中的经验教训。
原文摘要 · Abstract (English)
LLM agents increasingly rely on tool calling to act on external systems, and the Model Context Protocol (MCP) has quickly become its de facto interface. Operating MCP at cloud scale, however, becomes difficult. On the tool provider side, legacy services are not directly callable through MCP; the rapid protocol development also creates ongoing compatibility cost. On the agent side, the number of accessible tool is limited by the LLM context window and inference overhead; mounting a large tool set increases token usage and inference latency and can reduce task success rate. Moreover, for stateful MCP backends with multiple replicas, preserving session affinity increases client-side complexity. We present a cloud-scale gateway system for MCP service. It breaks the direct-connect model on the data plane and offloads legacy service integration, consolidating incompatible MCP variants, access control, tool recommendation, and session-aware routing to the gateway. Hybrid retrieval sustains 98% Top-15 recall; it scales agent tool access to 3,000+ with high tool selection accuracy, and reduces tool selection time by $8.9\times$ and token usage by $23.8\times$, with low per-call overhead, stable under scale-out. Finally, we share the lessons learned from deploying the gateway system in production.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。