自动构建评分标准，让大模型在开放问答中获得精准奖励。
ARES: Automated Rubric Synthesis for Scalable LLM Reinforcement Learning
- 从预训练文本自动生成带权重评分标准的问答对
- 构建10万条跨10个领域的标注数据，支持细粒度评估
- 适合需要多维度评价的开放任务，如医疗和指令遵循
基于评分标准的奖励机制为扩展大语言模型的强化学习提供了新路径，尤其适用于无法自动验证答案的任务。然而，现有方法依赖人工编写评分标准和问题集，且固定层级的评分标准难以适配具体问题需求。本文提出ARES（自动化评分标准生成框架），可规模化自动生成基于评分标准的强化学习数据。该框架从原始预训练文档出发，将知识转化为自包含的问答对，并协同生成针对每个问题的加权评分标准，实现对开放式回答的实例级奖励监督。为提升多样性和质量，ARES结合领域标签与角色信息进行生成控制，并引入校验过滤器，确保问题自洽性、答案忠实性及评分标准有效性。基于ARES，我们在10个领域构建了10万条带评分标准的样本。在7个基准测试上，使用ARES生成数据训练的评分标准强化学习模型，显著优于持续预训练、监督微调和二值奖励强化学习，在医疗、指令遵循等多维开放任务中表现最优。
原文摘要 · Abstract (English)
Rubric-based rewards offer a promising way to extend reinforcement learning (RL) for large language models beyond tasks with automatically verifiable answers. However, scaling rubric-based RL remains challenging: existing approaches often rely on expert-written rubrics and manually constructed question sets, while fixed task-level rubrics may fail to capture the evaluation requirements of individual questions. We propose ARES (Automated Rubric synthEsis for Scalable RL), a framework for automatically constructing rubric-based RL data at scale. Starting from raw pretraining documents, ARES converts source knowledge into self-contained question-answer pairs and co-generates question-specific weighted rubrics, enabling instance-level reward supervision for open-ended responses. To improve diversity and quality, ARES conditions generation on domain labels and persona information, and applies validation filters for question self-containment, answer faithfulness, and rubric validity. Using ARES, we construct 100K rubric-annotated instances across ten domains. Experiments on seven benchmarks show that rubric-based RL trained with ARES, outperforms continual pretraining, supervised fine-tuning, and binary-reward RL, with the largest gains on multi-dimensional open-ended tasks such as healthcare and instruction following.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。