让大模型评委用代码工具精准评估回答质量，性能超传统方法。
Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
- 引入代码执行器，让模型通过计算验证复杂约束
- 在7个基准上比强基线提升6.4%（点对点）和7.7%（成对）
- 无需人工标注数据，模型可自进化，适合高精度评估场景
大语言模型广泛用于评估回答质量，但多数仅依赖文本推理，难以验证复杂约束或精确计算。受工具增强推理（TIR）成功启发，我们提出TIR-Judge，一种端到端强化学习框架，集成代码执行器实现精准评估。该框架基于三大原则：（i）覆盖可验证与不可验证领域的多样化训练；（ii）支持点对点、成对、列表式判断格式；（iii）通过迭代强化学习直接从初始模型自举，无需知识蒸馏。在7个公开基准上，TIR-Judge在点对点评估中优于强基线6.4%，成对评估中提升7.7%，列表式表现接近Claude-Opus-4，且仅需80亿参数。令人瞩目的是，TIR-Judge-Zero（完全无蒸馏训练）达到与蒸馏版本相当的性能，证明工具增强的裁判模型可通过迭代强化学习自我演化。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are widely used as judges to evaluate response quality, providing a scalable alternative to human evaluation. However, most LLM judges operate solely on intrinsic text-based reasoning, limiting their ability to verify complex constraints or perform accurate computation. Motivated by the success of tool-integrated reasoning (TIR) in numerous tasks, we propose TIR-Judge, an end-to-end RL framework for training LLM judges that integrates a code executor for precise evaluation. TIR-Judge is built on three principles: (i) diverse training across verifiable and non-verifiable domains, (ii) flexible judgment formats (pointwise, pairwise, listwise), and (iii) iterative RL that bootstraps directly from the initial model without distillation. On seven public benchmarks, TIR-Judge surpasses strong reasoning-based judges by up to 6.4% (pointwise) and 7.7% (pairwise), and achieves listwise performance comparable to Claude-Opus-4 despite having only 8B parameters. Remarkably, TIR-Judge-Zero - trained entirely without distilled judge trajectories, matches the performance of distilled variants, demonstrating that tool-augmented judges can self-evolve through iterative reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。