评测智能体任务路由策略，发现动态组合决策更优
MetaRoute-Bench: Evaluating Meta-Decision Policies for Agentic Workflow Routing
- 构建统一评估框架，对比不同任务路由策略表现
- 动态组合策略成功率79.4%，优于静态策略2.7个百分点
- 适合研究智能体决策机制与系统优化的开发者
智能体在执行任务时需反复决策：直接回答、分解任务、调用工具、执行代码、委派专家、验证结果或故障恢复。这些元决策不仅影响任务成功率，还关乎运行成本和延迟，但常被嵌入编排框架中，仅以整体任务准确率评估。本文提出MetaRoute-Bench，一个开放可查的评估框架，用于在统一执行模型下比较元决策策略。初始基准包含180个合成任务（涵盖数据分析、研究、文档处理），8种路由策略，30组随机种子。共生成43,200条执行轨迹。结果显示，任务感知的组合策略成功率达79.4%，优于强工作负载特定静态策略的76.7%、一次性任务路由的67.4%以及直接回答的52.9%。相比静态策略，提升2.7个百分点（95%置信区间±2.0），代价为均值成本上升4.7%、延迟增加6.4%。消融实验表明，若路由组合仅限单一操作或移除验证环节，损失最大。所有结果基于预设种子的离线执行模型，不涉及真实部署；核心贡献在于可复现的评估方法与路由策略权衡分析。我们开源任务生成、策略、轨迹、测试与分析工具，支持真实系统验证。
原文摘要 · Abstract (English)
Agentic systems must repeatedly decide whether to answer directly, decompose a task, invoke a tool, execute code, delegate to a specialist, verify an intermediate result, or recover from failure. These meta-decisions affect not only task success but also operating cost and latency, yet they are often embedded inside an orchestration framework and evaluated only through aggregate task accuracy. We present MetaRoute-Bench, an open, inspectable framework for comparing meta-decision policies under a shared execution model. The initial benchmark contains 180 synthetic task profiles spanning data analysis, research, and document processing, eight routing policies, and 30 paired random seeds. Across 43,200 traces, a task-aware compositional policy achieves 79.4% success compared with 76.7% for a strong workload-specific static policy, 67.4% for one-shot task routing, and 52.9% for direct answering. Relative to the static policy, this is a 2.7 percentage-point improvement with paired 95% CI of plus or minus 2.0 points, at 4.7% higher mean cost and 6.4% higher latency. Ablations show the largest losses when route composition is restricted to one operation and when verification is removed. These results are generated by a seeded offline execution model rather than a live deployment; accordingly, the primary contribution is a reproducible evaluation method and an analysis of routing-policy tradeoffs, not evidence of production effectiveness. We release task generation, policies, traces, tests, and analysis artifacts to support live-system validation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。