让大模型当裁判时，推理未必划算，该用就用。
Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge
- 根据任务难易动态选是否启用推理，省成本
- 在分布偏移下仍保持高准确率，比固定策略好
- 适合需要高效评估的AI评测场景
具备推理能力的大语言模型（LLMs）被用于自动化评判，但其在评判中的收益与代价尚不明确。通过对比有无推理的评判者，我们发现显式推理能显著提升需结构化验证的任务（如数学和编程）的判断准确率，但在简单评估中效果有限甚至为负，且计算开销显著增加。这表明应有选择地使用推理而非全量启用，并需考虑潜在的分布偏移。为此，我们提出鲁棒自适应成本高效的路由方法（RACER），在固定预算下将路由问题建模为分布鲁棒优化问题，通过KL散度不确定集显式建模分布偏移，采用高效原始-对偶算法求解，具有最优策略唯一性和线性收敛的理论保证。大量实验表明，RACER在分布偏移下实现了更优的准确率-成本权衡。
原文摘要 · Abstract (English)
Reasoning-capable large language models (LLMs) have recently been adopted as automated judges, but their benefits and costs in LLM-as-a-Judge settings remain unclear. Through controlled comparisons between reasoning and non-reasoning judges, we show that explicit reasoning substantially improves judgment accuracy on tasks requiring structured verification (e.g., math and coding), while offering limited or even negative gains on simpler evaluations and incurring significantly higher computational cost. These findings motivate that reasoning should be used selectively rather than universally, with awareness of possible distribution shift. We propose a Robust Adaptive Cost-Efficient Routing (RACER), which dynamically selects between reasoning and non-reasoning judges under a fixed budget by formulating routing as a constrained distributionally robust optimization problem. RACER explicitly accounts for distribution shift via a KL-divergence uncertainty set, admits an efficient primal--dual algorithm, and enjoys theoretical guarantees including uniqueness of the optimal policy and linear convergence. Extensive experiments show that RACER achieves superior accuracy--cost trade-offs under distribution shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。