用评分标准提升大模型生成质量,让AI写作更像人。
Reinforcement Learning with Rubric Anchors
- 用人工与AI协作设计超1万条评分标准,自动评估开放任务输出。
- 仅需5000样本就提升5.2%表现,优于6710亿参数模型。
- 可精准控制风格,减少机械感,适合内容创作与教育应用。
强化学习从可验证奖励(RLVR)已成为提升大语言模型能力的重要范式,如OpenAI的o系列模型所示。在RLVR中,奖励来自可验证信号——例如代码生成中的通过单元测试或数学推理中的正确答案匹配。然而,这一要求将应用范围局限于可自动验证的任务。为突破此限制,我们拓展了RLVR范式至开放性任务,引入基于评分标准的奖励机制:精心设计的评分标准作为结构化、模型可解释的评判准则,实现对主观输出的自动评分。我们构建了迄今最大的评分标准奖励系统,包含超过10,000条由人类、大模型或人机协同生成的标准。针对基于评分标准的强化学习实施难题,我们提出清晰框架,并发布开源的Qwen-30B-A3B模型,取得显著成效:1)仅使用5000余样本,系统在开放性基准上提升5.2%,尤其在人文类任务中表现突出,超越6710亿参数的DeepSeek-V3模型2.4%;同时保持通用与推理能力。2)该方法提供细粒度风格控制,利用评分标准作为锚点,缓解‘人工智能式’语气,生成更具人类特征、更富表现力的回答。我们分享了评分标准构建、数据筛选与训练的关键经验,并讨论当前局限与未来计划。
原文摘要 · Abstract (English)
Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a powerful paradigm for enhancing Large Language Models (LLMs), exemplified by the success of OpenAI's o-series. In RLVR, rewards are derived from verifiable signals-such as passing unit tests in code generation or matching correct answers in mathematical reasoning. While effective, this requirement largely confines RLVR to domains with automatically checkable outcomes. To overcome this, we extend the RLVR paradigm to open-ended tasks by integrating rubric-based rewards, where carefully designed rubrics serve as structured, model-interpretable criteria for automatic scoring of subjective outputs. We construct, to our knowledge, the largest rubric reward system to date, with over 10,000 rubrics from humans, LLMs, or a hybrid human-LLM collaboration. Implementing rubric-based RL is challenging; we tackle these issues with a clear framework and present an open-sourced Qwen-30B-A3B model with notable gains: 1) With only 5K+ samples, our system improves by +5.2% on open-ended benchmarks (especially humanities), outperforming a 671B DeepSeek-V3 model by +2.4%, while preserving general and reasoning abilities. 2) Our method provides fine-grained stylistic control, using rubrics as anchors to mitigate the "AI-like" tone and produce more human-like, expressive responses. We share key lessons in rubric construction, data selection, and training, and discuss limitations and future releases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。