用250万数据训练出高效推理评估器,开源模型超越70B专有模型。
Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
- 用250万样本迭代训练,不依赖强化学习,直接提升评估精度。
- 200亿参数评估器在MATH任务中接近最优表现,比传统方法提升14.1%。
- 适合需要高可靠评估的推理型模型研发者与评测团队使用。
微调专用生成式评估器已成为满足训练与测试阶段可扩展评估需求的主流范式。然而,现有工作多集中于新方法(如强化学习)的应用,忽视了大规模数据驱动的发展路径。本文聚焦数据规模,构建了涵盖五个独立评估任务(成对比较、步骤级评估、无参考/有参考验证、单评分)及多个推理导向领域的250万样本数据集。基于此数据,我们训练了基础自动推理评估器(FARE),包括80亿和200亿参数版本(其中36亿为活跃参数),采用简单的迭代拒绝采样监督微调(SFT)方法。FARE-8B挑战了更大的专用强化学习训练评估器,而FARE-20B成为开源评估器的新标准,性能超越专用700亿以上模型。在真实场景中,作为推理时重排序器,FARE-20B在MATH任务上达到近似最优表现;作为强化学习训练中的验证器,使下游模型性能相比字符串匹配验证器提升最高达14.1%;从FARE初始化的持续微调版FARE-Code,在测试用例质量评估上优于gpt-oss-20B达65%。
原文摘要 · Abstract (English)
Finetuning specialized generative evaluators has emerged as a popular paradigm to meet the increasing demand for scalable evaluation during both training and test-time. However, recent work has largely focused on applying new methodology, such as reinforcement learning (RL), to training evaluators, shying away from large-scale, data-driven development. In this work, we focus on data scaling, curating a set of 2.5M samples spanning five unique evaluation tasks (pairwise, step-level, reference-free and reference-based verification, and single rating) and multiple domains focused on reasoning evaluation. With our data, we train Foundational Automatic Reasoning Evaluators (FARE), a family of 8B and 20B (with 3.6B active) parameter evaluators, with a simple iterative rejection-sampling supervised finetuning (SFT) approach. FARE-8B challenges larger specialized RL-trained evaluators and FARE-20B sets the new standard for open-source evaluators, surpassing specialized 70B+ evaluators. Beyond static benchmarks, we evaluate FARE in real-world tasks: As inference-time rerankers, FARE-20B achieves near-oracle performance on MATH. As verifiers in RL training, FARE improves the downstream RL-trained model performance by up to 14.1% vs. string-matching verifiers. When initialized from FARE, a continually-finetuned FARE-Code outperforms gpt-oss-20B by 65% on evaluating test-case quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。