arXiv:2606.28925cs.LGcs.AI2026-06

提出多智能体路由新基准,评估选哪个+多少个智能体最划算。

Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation

论文配图:Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation
图 1 · 摘自论文原文
  • 把路由问题看作选多个智能体的集合预测,用真实数据标注。
  • 精细模型比零样本和最近邻方法准确率高,加成本权重后更省资源。
  • 适合研究智能体协作效率、成本控制或构建复杂工具链的人。

从自然语言指令中进行工具与智能体路由本质上是集合值预测问题:单个请求可能需要多个智能体,但过多选择会增加执行成本。本文基于WildChat构建了包含3,000个提示的基准,固定12个智能体目录,采用AI辅助启发式标注与固定标注模式,并对多标签评估进行受控重平衡。评估协议融合集合级指标(精确率、召回率、F1、Jaccard、精确匹配)、延迟、面向执行的能力覆盖模拟,以及基于序数智能体成本层级的约束加权路由设置。对比方法包括最近邻匹配、线性多标签分类、依赖感知基线、微调编码器、确定性加权后评分(WAR)及零样本大模型基线。结果表明,监督式路由显著优于最近邻与零样本大模型。微调编码器在无约束条件下达到最高集合准确率,线性多标签模型则提供最强实用基线。在约束设置下,加权路由层能提升强监督评分器的效用,尤其在编码器+WAR组合中增益最大。整体上,该基准与评估协议支持可复现的固定目录多智能体路由中准确率-成本权衡研究。

原文摘要 · Abstract (English)

Tool and agent routing from natural-language prompts is naturally a set-valued prediction problem: a single query may require multiple agents, while over-selection increases execution cost. The benchmark introduced here is derived from WildChat and contains 3,000 prompts over a fixed 12-agent catalog, with AI-assisted heuristic labels under a fixed schema and controlled rebalancing for multi-label evaluation. The evaluation protocol combines set-level metrics (Precision, Recall, F1, Jaccard, and Exact Match), latency, an execution-oriented capability-coverage simulation, and a constrained weighted-routing setting based on ordinal agent-cost tiers. Compared methods include nearest-neighbor matching, linear multilabel classification, dependency-aware baselines, a fine-tuned encoder, deterministic weighted post-scoring via Weighted Agent Routing (WAR), and a zero-shot LLM baseline. Results show that supervised routers substantially outperform nearest-neighbor and zero-shot LLM routing. The fine-tuned encoder achieves the strongest unconstrained set accuracy, while the linear multilabel model provides the strongest practical baseline. In the constrained setting, the weighted routing layer improves utility when applied on top of strong supervised scorers, with the largest gain observed for Encoder+WAR. Overall, the benchmark and evaluation protocol support reproducible study of accuracy-cost trade-offs in fixed-catalog multi-agent routing.

多智能体路由优化评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。