arXiv:2605.18859cs.LGcs.AI2026-05被引 4

提出双轨评测框架,实现智能路由的快速验证与真实应用测试。

TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing

论文配图:TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing
图 1 · 摘自论文原文
  • 构建分步路由评测数据集,支持静态与动态两种评估模式。
  • 在SWE-bench等5个数据集上测试,100例独立验证中成功率达92.3%。
  • 无需在线大模型评分,可快速迭代,适合研究高效代理路由系统者。

长周期应用如编程代理、深度研究系统和计算机使用代理中,单个用户请求会触发多次模型调用。将每次调用路由至成本最低但足够准确的模型,可在不牺牲质量的前提下降低成本。然而现有路由评测仅针对单次提示,未暴露中间步骤的前置文本,未检验廉价替代模型是否保持下游任务成功率,且常依赖在线大模型评判。本文提出TwinRouterBench,一个分步级路由评测基准,包含两个赛道:静态赛道提供来自SWE-bench、BFCL、mtRAG、QMSum和PinchBench的970个路由器可见前缀,覆盖520个实例,每个均配执行验证的目标层级(基于释放的降级-级联协议),评分基于层级标签、轨迹归属与令牌成本的确定性算术运算,无需在线大模型评判;动态赛道提供运行路由器的工具包,在完整的500例SWE-bench Verified套件上运行,本文报告了100例与静态训练集无交集的保留评估结果。每轮模型调用,路由器从固定池中选择具体模型,成功与否由官方任务完成情况与实际API支出共同判定。两个赛道支持快速离线迭代与真实代理执行下的端到端验证。代码与数据已公开于https://github.com/CommonstackAI/TwinRouterBench。

原文摘要 · Abstract (English)

LLM routing matters most in long-horizon applications such as coding agents, deep research systems, and computer-use agents, where a single user request triggers many model calls. Routing each call to the cheapest sufficient model can cut costs without sacrificing quality, yet existing router benchmarks evaluate routers only on one-shot prompts. They never expose the router-visible prefix at an intermediate agent step, never test whether a cheaper replacement preserves downstream task success, and often rely on online LLM judges at evaluation time. We introduce TwinRouterBench, a step-level routing benchmark with two tracks. The static track provides 970 router-visible prefixes from 520 instances across SWE-bench, BFCL, mtRAG, QMSum, and PinchBench, each paired with an execution-verified target tier estimated under a released downgrade-and-cascade protocol; scoring is deterministic arithmetic over tier labels, trajectory membership, and token costs, with no online evaluator-side LLM judge. The dynamic track supplies a harness that runs routers on the full 500-case SWE-bench Verified suite; in this paper we report a 100-case held-out evaluation disjoint from the static SWE supervision split. At each LLM call the router selects a concrete model from a locked pool, and success is measured by official task resolution and realized API spend. The two tracks support fast offline iteration followed by end-to-end validation under live agent execution. Code and data are available at https://github.com/CommonstackAI/TwinRouterBench.

大模型路由评测基准代理系统成本优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。