构建大规模模型路由评测基准，揭示主流方法效果有限。
LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing
- 统一框架整合33个模型、21数据集，超40万条测试样本。
- 多数路由方法表现相近，商业方案未显著优于基础策略。
- 适合研究模型选择与资源效率的学者，尤其关注推理优化者。
大型语言模型（LLM）路由旨在为每个查询从模型集合中分配最合适的模型。本文提出LLMRouterBench，一个大规模评测基准与统一框架。该基准包含来自21个数据集的超过40万条实例，覆盖33个模型，并提供性能导向与性能-成本权衡两类评估指标，集成10种代表性路由基线。基于此，我们系统重评了该领域：虽确认模型间存在显著互补性（路由的核心前提），但发现多数路由方法在统一评估下表现接近，若干近期方法（包括商用方案）未能稳定超越简单基线。同时，与理想最优（Oracle）仍存在显著差距，主要源于持续存在的模型召回失败。此外，实验表明骨干嵌入模型影响有限，更大模型集合带来的收益递减，而精心筛选模型更优；基准还支持延迟敏感分析。所有代码与数据已开源于https://github.com/ynulihao/LLMRouterBench。
原文摘要 · Abstract (English)
Large language model (LLM) routing assigns each query to the most suitable model from an ensemble. We introduce LLMRouterBench, a large-scale benchmark and unified framework for LLM routing. It comprises over 400K instances from 21 datasets and 33 models. Moreover, it provides comprehensive metrics for both performance-oriented routing and performance-cost trade-off routing, and integrates 10 representative routing baselines. Using LLMRouterBench, we systematically re-evaluate the field. While confirming strong model complementarity-the central premise of LLM routing-we find that many routing methods exhibit similar performance under unified evaluation, and several recent approaches, including commercial routers, fail to reliably outperform a simple baseline. Meanwhile, a substantial gap remains to the Oracle, driven primarily by persistent model-recall failures. We further show that backbone embedding models have limited impact, that larger ensembles exhibit diminishing returns compared to careful model curation, and that the benchmark also enables latency-aware analysis. All code and data are available at https://github.com/ynulihao/LLMRouterBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。