用操作系统调度思路解决多个AI编程代理并发调用API时的失败问题
HiveMind: OS-Inspired Scheduling for Concurrent LLM Agent Workloads
- 借鉴操作系统机制,用五种调度策略协调多代理请求
- 并发50个代理时失败率从100%降到18%,浪费算力减少超90%
- 无需改代码,可对接主流模型接口,适合部署AI工作流的团队
当多个LLM编程代理共享一个速率受限的API端点时,其资源争用模式类似于未调度的操作系统进程竞争CPU、内存和I/O。在一次典型案例中,11个并行代理中有3个因连接重置和HTTP 502错误失败,失败率达27%,尽管该API的总容量足以顺序服务全部11个代理。本文提出HIVEMIND,一种透明的HTTP代理,采用五种操作系统启发式调度原语——准入控制、速率限制追踪、基于AIMD的背压与熔断机制、令牌预算管理、优先级队列——以消除因无协调并行执行导致的故障模式。该代理无需修改现有代理代码,通过自动检测提供者配置支持Anthropic、OpenAI及本地模型API。在七种场景(5-50个并发代理)下的评估显示,未经协调的代理在资源竞争下失败率为72%-100%,而HIVEMIND将失败率降至0%-18%,并消除48%-100%的浪费计算。消融实验表明,透明重试是单一最关键的机制,但各原语组合效果最佳。真实世界验证表明,相较于Ollama,HIVEMIND每请求增加的代理开销低于3ms。系统已开源,采用MIT许可证。
原文摘要 · Abstract (English)
When multiple LLM coding agents share a rate-limited API endpoint, they exhibit resource contention patterns analogous to unscheduled OS processes competing for CPU, memory, and I/O. In a motivating incident, 3 of 11 parallel agents died from connection resets and HTTP 502 errors - a 27% failure rate - despite the API having sufficient aggregate capacity to serve all 11 sequentially. We present HIVEMIND, a transparent HTTP proxy that applies five OS-inspired scheduling primitives - admission control, rate-limit tracking, AIMD backpressure with circuit breaking, token budget management, and priority queuing - to eliminate the failure modes caused by uncoordinated parallel execution. The proxy requires zero modifications to existing agent code and supports Anthropic, OpenAI, and local model APIs via auto-detected provider profiles. Our evaluation across seven scenarios (5-50 concurrent agents) shows that uncoordinated agents fail at 72-100% rates under contention, while HIVEMIND reduces failures to 0-18% and eliminates 48-100% of wasted compute. An ablation study reveals that transparent retry - not admission control - is the single most critical primitive, but the primitives are most effective in combination. Real-world validation against Ollama confirms that HIVEMIND adds under 3ms of proxy overhead per request. The system is open-source under the MIT license.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。