通过筛选低熵词元,提升多轮自蒸馏的稳定性与准确率。
SMOPD: Selective Token-Entropy Masking for Dirty-History Multi-Turn On-Policy Self-Distillation
- 按学生模型输出的词元熵值筛选,剔除20%低熵词元以稳定训练。
- 在Qwen3模型上,1.7B至8B规模下提升1.0-2.5个百分点准确率。
- 无需额外参数和推理开销,适合追求高效多轮对话训练的研究者。
脏历史回放使多轮在线自蒸馏(OPSD)变得脆弱:一旦学生模型生成错误中间回复,后续轮次将依赖该错误,统一蒸馏可能浪费损失在修正信号微弱的词元上。本文提出SMOPD(选择性掩码用于在线蒸馏),一种仅修改损失函数的稳定方法。对每个中间回复,按学生模型熵值排序词元位置,剔除最低熵20%的词元后进行截断广义Jensen-Shannon蒸馏;最终答案与全保留损失不变。该设计聚焦词元级不确定性,不引入参数,且推理无开销。在LiC数据集上使用Qwen3模型测试,1.7B、4B和8B单种子实验中,SMOPD使SHARDED视图准确率提升1.0-2.5个百分点;4B多种子验证显示平均提升+1.7pp(双尾p=0.022)。引入结果标量虽在4B下增益+1.3pp,但在1.7B下减损-4.0pp,且组合效果具有规模依赖性。结果表明,在该设定下,词元级不确定性比最终答案正确性更可靠,但因果机制与泛化验证仍需后续工作。
原文摘要 · Abstract (English)
Dirty-history rollouts make multi-turn on-policy self-distillation (OPSD) brittle: once a student emits an erroneous intermediate reply, later turns are conditioned on that reply, and uniform distillation can spend loss on tokens that carry little corrective signal. We introduce SMOPD (Selective Masking for On-Policy Distillation), a loss-only stabilization method for multi-turn OPSD. For each generated middle-turn reply, SMOPD ranks token positions by student entropy and removes the lowest-entropy 20% from the clipped generalized Jensen-Shannon distillation loss; final-answer and FULL-preservation losses are unchanged. This design targets token-level uncertainty rather than coarse trajectory outcomes, adds no parameters, and has zero inference-time overhead. We compare SMOPD with a correctness-scaling variant that multiplies a common detached reliability proxy using final-answer correctness. On LiC with Qwen3 models, SMOPD improves SHARDED-view accuracy by 1.0-2.5 percentage points in single-seed 1.7B, 4B, and 8B comparisons, and a small 4B multi-seed check shows a +1.7pp mean SHARDED gain over baseline (two-tailed p = 0.022). Adding the outcome scalar is harmful without masking at 1.7B (-4.0pp) and remains scale-dependent when combined with masking (+1.3pp at 4B, neutral at 1.7B, and -0.5pp at 8B). These archived aggregate results suggest that token-level uncertainty is a more reliable stabilization signal than scalar final-answer correctness in this evaluated dirty-history OPSD setting, while leaving causal mechanism tests and broader benchmark validation to future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。