发现大模型路由评估存在严重偏差,导致性能天花板被高估。
Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts
- 通过双评判与精确匹配验证,识别出评估偏差来源
- 实测未解决率因评测问题被夸大,部分任务下降超13个百分点
- 适合关注多模型路由效率与评测可信度的研究者
高效跨多大模型路由可通过将请求分配给最便宜的可用模型实现成本-质量权衡。以往研究将路由潜力限制归因于‘不可解上限’——即池中无模型能解决的问题。我们基于Gemma 4和Llama 3.1系列,在六项基准(MMLU、MedQA、HumanEval、MBPP、Alpaca、ShareGPT)上对206,000个查询-模型对进行大规模研究,采用LLM作为裁判和精确匹配指标。结果表明,大量报告的‘不可解’现象源于评估偏差:(i)裁判系统偏好冗长而非正确输出,(ii)固定生成预算下的截断,(iii)输出格式不一致。通过双裁判验证和精确匹配锚定,我们显著降低了各任务上的测量不可解率。提出分解框架,揭示这些偏差在不同领域和模型族中具有稳定模式。这些偏差也扭曲了路由器训练信号:标准路由器退化为多数类预测(约79%最小层级最优),经随机特征与标签打乱控制确认,造成13–17个百分点的机会损失。提出双裁判验证、精确匹配锚定及成本敏感目标等建议。研究显示现有路由潜力估计严重虚高,强调多大模型系统需建立可靠评估协议。
原文摘要 · Abstract (English)
Efficient routing across multiple LLMs enables cost-quality tradeoffs by directing queries to the cheapest capable model. Prior work attributes routing headroom to an "unsolvability ceiling", queries no model in the pool can solve. We present a large-scale study of multi-tier LLM routing with 206,000 query-model pairs across six benchmarks (MMLU, MedQA, HumanEval, MBPP, Alpaca, ShareGPT) using the Gemma 4 and Llama 3.1 families. Evaluating with both LLM-as-a-judge and exact-match metrics, we show that a substantial portion of reported unsolvability stems from evaluation artifacts: (i) systematic judge biases favoring verbosity over correctness, (ii) truncation under fixed generation budgets, and (iii) output format mismatches. Through dual-judge validation and exact-match grounding, we reduce measured unsolvability across tasks. We introduce a decomposition framework attributing failures to these artifacts, revealing consistent patterns across domains and model families. These artifacts also distort router training signals: standard routers collapse to majority-class prediction (~79% smallest-tier optimal), confirmed via random-feature and shuffled-label controls, incurring a 13-17 percentage point opportunity cost. We provide actionable recommendations including dual-judge validation, exact-match anchoring, and cost-sensitive objectives. Our findings suggest existing routing headroom estimates are substantially inflated, underscoring the need for reliable evaluation protocols in multi-LLM systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。