用测试时扩展提升大模型评分器性能,让判断更准且可解释。
J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge
- 通过强化学习训练模型,使其在测试时扩展计算能持续提效。
- 相比之前最优模型,评分准确率提升4.8%,扩展趋势强5.1%。
- 首次发现强化学习赋予模型真正测试时扩展能力,适合评测研究者。
当前人工智能研究重心正从模型训练转向评估质量提升,这对推动系统进一步发展至关重要。传统评估依赖奖励模型给出单一评分,虽有效但缺乏可解释性。大模型作为评判者(LLM-as-a-Judge)提供了更可扩展、可解释的监督方式,并能提供推理轨迹。随着大推理模型兴起,其需更多token进行深度思考与答案优化,因此在LLM-as-a-Judge范式中扩大测试时计算量成为进一步提升性能和可解释性的关键路径。本文提出J1-7B:先在拒绝采样收集的反思增强数据集上微调,再通过具备可验证奖励的强化学习训练。推理时应用简单测试时扩展(STTS)策略以提升性能。实验表明,J1-7B相较此前最先进水平提升4.8%,在STTS下展现出5.1%更强的扩展趋势。此外,我们发现:(1) 现有LLM-as-a-Judge本身不具备此扩展趋势;(2) 仅经反思数据微调的模型仍表现弱扩展;(3) 显著扩展趋势主要出现在强化学习阶段,说明有效的测试时扩展能力主要由强化学习获得。
原文摘要 · Abstract (English)
The current focus of AI research is shifting from emphasizing model training towards enhancing evaluation quality, a transition that is crucial for driving further advancements in AI systems. Traditional evaluation methods typically rely on reward models assigning scalar preference scores to outputs. Although effective, such approaches lack interpretability, leaving users often uncertain about why a reward model rates a particular response as high or low. The advent of LLM-as-a-Judge provides a more scalable and interpretable method of supervision, offering insights into the decision-making process. Moreover, with the emergence of large reasoning models, which consume more tokens for deeper thinking and answer refinement, scaling test-time computation in the LLM-as-a-Judge paradigm presents an avenue for further boosting performance and providing more interpretability through reasoning traces. In this paper, we introduce $\textbf{J1-7B}$, which is first supervised fine-tuned on reflection-enhanced datasets collected via rejection-sampling and subsequently trained using Reinforcement Learning (RL) with verifiable rewards. At inference time, we apply Simple Test-Time Scaling (STTS) strategies for additional performance improvement. Experimental results demonstrate that $\textbf{J1-7B}$ surpasses the previous state-of-the-art LLM-as-a-Judge by $ \textbf{4.8}$\% and exhibits a $ \textbf{5.1}$\% stronger scaling trend under STTS. Additionally, we present three key findings: (1) Existing LLM-as-a-Judge does not inherently exhibit such scaling trend. (2) Model simply fine-tuned on reflection-enhanced datasets continues to demonstrate similarly weak scaling behavior. (3) Significant scaling trend emerges primarily during the RL phase, suggesting that effective STTS capability is acquired predominantly through RL training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。