arXiv:2606.01476cs.LGcs.CL2026-06被引 3

不用教师的逐词概率,用语义块验证提升学生模型训练效果。

OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification

论文配图:OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification
图 1 · 摘自论文原文
  • 用多词块的语义相似度替代逐词概率匹配,实现无需教师输出的训练。
  • 在高不确定性的推理分支处集中监督,数学任务性能提升28.64%。
  • 适合使用闭源强模型做教师,可超越自探索强化学习表现。

在策略蒸馏(OPD)中,学生模型基于教师提供的密集词级反馈进行训练,避免了监督微调的分布偏移和强化学习的稀疏奖励问题。然而传统OPD存在两个耦合缺陷:一需访问教师的词级概率输出,排除大量闭源强模型;二词级概率信号脆弱,依赖教师与学生间下一词的狭窄重叠,易放大重复等退化模式。本文提出OmniOPD,通过无词级概率、基于语义块的连续监督机制解决上述问题。OmniOPD采用蒙特卡洛推演,以多词块间的语义相似度近似教师局部偏好,并结合峰值熵调度器仅在学生高不确定性分支处审计。引入狄利克雷-多项式贝叶斯先验与基础模型KL锚点,控制离散采样方差,防止未审核词的策略坍缩。在多个基准测试中,OmniOPD相较标准OPD在数学任务上最高提升28.64%。当搭配Claude-4.5-Haiku和Gemini-2.5-Flash等闭源强教师时,较其开源教师版本再提升9.54%,推动学生模型超越自探索强化学习表现。

原文摘要 · Abstract (English)

On-Policy Distillation (OPD) trains a student model on its own generative trajectories under dense token-level feedback from a stronger teacher, mitigating both the off-policy distribution shift of Supervised Fine-Tuning (SFT) and the sparse credit assignment of Reinforcement Learning (RL). However, standard OPD faces two coupled limitations. First, it requires direct access to the teacher's token-level logits, excluding a broad class of capable proprietary models from serving as teachers. Second, the token-level logit signal itself is brittle, depending on a narrow overlap of plausible next tokens between teacher and student, and prone to amplifying degenerate patterns such as repetition loops. In this paper, we introduce OmniOPD, a novel framework that addresses both limitations through a logit-free, chunk-level supervision signal. OmniOPD replaces deterministic logit matching with Monte Carlo rollouts that approximate the teacher's local preferences through a continuous semantic similarity metric over multi-token chunks, and concentrates this supervision via a peak-entropy scheduler that audits the student only at its high-uncertainty reasoning forks. A Dirichlet-Multinomial Bayesian prior and a base-model KL anchor further bound the variance of discrete sampling and prevent policy collapse across unaudited tokens. Across competitive benchmarks, OmniOPD surpasses the standard OPD approach by up to +28.64% on math, confirming that chunk-level semantic verification extracts a more reliable learning signal than token-level logit matching, whose high information density is offset by significant noise and brittleness. Furthermore, when paired with stronger black-box teachers such as Claude-4.5-Haiku and Gemini-2.5-Flash, OmniOPD achieves an additional +9.54% relative on math over its open-weight teacher counterpart, advancing the student past the performance of self-exploratory RL.

模型蒸馏强化学习语义验证闭源模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。