arXiv:2608.02948cs.LGcs.AI2026-08

用评分标准指导生成,让模型产出更符合预期的开放式回答。

Rubrics as Privileged Information for Open-Ended Generation

  • 用评分标准代替标准答案作为教学信号,引导模型生成多样化有效回复。
  • 在HealthBench上比强化学习方法高0.10分,优于参考答案蒸馏法0.034~0.079分。
  • 适用于医疗、科研等需高质量开放生成的场景,尤其适合有明确评价标准的任务。

基于策略的自蒸馏(OPSD)在数学等可验证领域表现良好,依赖真实答案这类硬性特权信息(PI)来约束生成结果。本文将OPSD拓展至开放式生成任务,采用软性特权信息——即评分标准(rubrics),以指导生成偏好但允许多种合理输出。相比传统强化学习中仅作为标量奖励的评分标准,本文证明其作为密集型PI能提供更丰富有效的训练信号。与硬性参考答案相比,软性评分标准不会过度约束模型,反而通过捕捉一组有效回复共有的偏好结构提升性能。实验表明,在Qwen和Llama系列模型上,该方法在HealthBench基准上超越了评分标准作为奖励的强化学习(RaR RL),绝对得分提升最高达+0.10;在相同训练设置下,优于参考答案蒸馏法+0.034至+0.079。进一步在RubricHub Science语料上训练并评估于ResearchQA任务,软性评分标准蒸馏法取得66.6%准确率,显著高于参考答案蒸馏(64.2%)和RaR RL(57.6%)。

原文摘要 · Abstract (English)

On-policy self-distillation (OPSD), where a single model acts as both student and teacher with different contexts, has shown promise in verifiable domains like math, where hard privileged information (PI) in the form of ground-truth answers structurally constrains valid continuations. We extend OPSD to open-ended generation using soft PI in the form of rubrics that guide preferences but admit many valid responses. Rubrics have served as scalar rewards for reinforcement learning (RL); we show that they provide substantially richer signal as dense PI for distillation, and contrary to intuition, soft rubric PI provides a larger and more effective training signal on student roll-outs than hard reference completion PI in this regime. A reference completion is one point in a set of valid responses, so distilling towards it over-constrains the student, while rubrics specify the preference structure shared across the set of valid responses. We show the effectiveness of using rubrics as PI for open-ended generation across Qwen and Llama model families and show that it outperforms rubric-as-reward (RaR) RL using HealthBench, a benchmark that grades open-ended health responses against physician-created rubrics, providing dense token-level supervision for open-ended tasks; RuPI beats RaR RL by up to +0.10 absolute score and, under matched recipe and KL direction, beats reference-PI by +0.034 to +0.079 absolute score across three models. We further show that these findings generalize to training on the RubricHub Science corpus and evaluating on ResearchQA: soft rubric PI outperforms both reference-PI distillation and RaR RL (66.6% vs. 64.2% and 57.6%).

开放生成评分标准自蒸馏强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。