用人类反馈数据训练模型，实现开放领域任务的推理时扩展，性能超越主流模型。
HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Tasks
- 三阶段流程：生成初稿→反馈模型提建议→编辑模型优化回复
- 在Arena Hard上达到92.7分，优于o1-preview和DeepSeek R1
- 适用于无标准答案的开放领域任务，适合需要持续改进的场景
推理时扩展是OpenAI o1和DeepSeek R1等模型成功的关键。然而，现有方法多依赖可验证答案的任务（如数学、编程），限制了其在开放领域应用。受人类试错与反馈机制启发，我们构建了HelpSteer3数据集，用于训练专门的反馈与编辑模型，实现开放领域任务的推理时扩展。系统中，首个模型生成初始响应，第二个模型提供详细反馈，第三个模型据此修改回应。实验表明，增加初始草稿数、反馈量及编辑次数可显著提升性能。当优化后，基于Llama 3 70B模型的系统在2025年3月5日达到Arena Hard 92.7分，超越OpenAI o1-preview-2024-09-12的90.4分和DeepSeek R1的92.3分。
原文摘要 · Abstract (English)
Inference-Time Scaling has been critical to the success of recent models such as OpenAI o1 and DeepSeek R1. However, many techniques used to train models for inference-time scaling require tasks to have answers that can be verified, limiting their application to domains such as math, coding and logical reasoning. We take inspiration from how humans make first attempts, ask for detailed feedback from others and make improvements based on such feedback across a wide spectrum of open-ended endeavors. To this end, we collect HelpSteer3 data to train dedicated Feedback and Edit Models that are capable of performing inference-time scaling for open-ended general-domain tasks. In our setup, one model generates an initial response, which are given feedback by a second model, that are then used by a third model to edit the response. We show that performance on Arena Hard, a benchmark strongly predictive of Chatbot Arena Elo can be boosted by scaling the number of initial response drafts, effective feedback and edited responses. When scaled optimally, our setup based on 70B models from the Llama 3 family can reach SoTA performance on Arena Hard at 92.7 as of 5 Mar 2025, surpassing OpenAI o1-preview-2024-09-12 with 90.4 and DeepSeek R1 with 92.3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。