arXiv:2606.19327cs.AIcs.CL2026-06被引 6

用评分标准指导推理模型自蒸馏,提升逻辑生成质量

Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation

论文配图:Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
图 1 · 摘自论文原文
  • 以评分标准为细粒度反馈,指导模型自蒸馏
  • 在多个科学推理任务上优于现有方法1.0~0.9分
  • 适合需要高质量推理链的场景,如教育评估

推理型语言模型的后训练通常依赖监督蒸馏和基于可验证奖励的强化学习。蒸馏常依赖代价高昂且可能不完整或有噪声的思维链标注;即使最终答案正确,不完整的推理过程也会干扰学习。而基于验证奖励的强化学习通常将评估反馈压缩为标量信号,掩盖了需改进的具体方面。本文提出鲁棒评分标准条件自蒸馏(Rubric-Conditioned Self-Distillation),利用结构化细粒度评分标准作为在线策略自蒸馏的指导信号。该方法让教师模型根据评分标准条件生成,为学生模型采样的推理轨迹提供逐标记级指导,避免将单一参考推理链视为唯一监督目标。评分标准定义了优秀回答应满足的条件,实现比标量奖励优化更精细的信用分配。我们采用两阶段流程:先学习生成任务特定评分标准,再训练评分引导的推理器。在多样化的科学推理基准测试中,结果表明该方法能有效将评分标准转化为推理过程中的逐标记指导,在平均性能上超越GRPO 1.0分、OPSD 0.9分。

原文摘要 · Abstract (English)

Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards. Distillation often relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect; even when the final solution is correct, an imperfect rationale can interfere with learning. Reinforcement learning with verified rewards, on the other hand, typically compresses evaluative feedback into a scalar signal, obscuring which aspects of a response should be improved. We propose \textbf{Rubric-Conditioned Self-Distillation}, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation. Our method conditions the teacher model on criterion-level rubrics and uses it to provide token-level guidance on the student's own sampled trajectories. This design avoids treating a single reference rationale as the sole supervision target. Instead, rubrics specify what a strong response should satisfy, enabling more fine-grained credit assignment over the reasoning process than scalar reward optimization. We instantiate this framework with a two-stage pipeline that first learns to generate task-specific rubrics and then trains a rubric-guided reasoner. We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance over the reasoning process, surpassing GRPO by 1.0 points and OPSD by 0.9 points on average.

自蒸馏推理模型评分标准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。