用评分标准替代教师输出，实现黑盒场景下的高效模型对齐
Rubric-based On-policy Distillation
- 通过师生对比生成特定提示的评分标准
- 在多数场景超越传统方法，样本效率提升10倍
- 适合私有模型或无法获取中间输出的对齐任务
基于策略的蒸馏（OPD）是模型对齐的有效方法，但依赖教师模型的输出概率分布，仅适用于白盒场景。本文提出使用结构化语义评分标准作为替代方案，使OPD仅需教师生成的文本响应即可运行。为此，我们设计了ROPD框架：从教师与学生输出的对比中提取特定提示的评分标准，并用于评估学生生成序列，实现在线优化。实验表明，ROPD在多数场景下优于先进基于输出概率的OPD方法，样本效率最高提升10倍。该结果证明，基于评分标准的OPD是一种灵活、兼容黑盒的替代方案，为私有和开源大模型的可扩展蒸馏提供简单而强大的基线。代码已公开于 https://github.com/Peregrine123/ROPD_official。
原文摘要 · Abstract (English)
On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios. We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses. To prove it, we introduce ROPD, a simple yet foundational framework for rubric-based OPD. Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization. Empirically, ROPD outperforms the advanced logit-based OPD methods across most scenarios, and achieving up to a 10x gain in sample efficiency. These results position rubric-based OPD as a flexible, black-box-compatible alternative to the prevailing logit-based OPD, offering a simple yet strong baseline for scalable distillation across proprietary and open-source LLMs. Code is available at https://github.com/Peregrine123/ROPD_official.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。