通过动态筛选高困惑度词元，让模型更专注逻辑推理而非模仿风格。
Robust Reasoning via Dynamic Token Selection for Distribution-Aligned Self-Distillation
- 用参考模型生成候选词元，再按基础模型置信度动态过滤
- 在数学、代码等任务上显著减少高困惑度词元，提升鲁棒性
- 适合追求推理能力增强的模型优化者，尤其对复杂任务有效
自蒸馏通过重写参考答案作为更匹配模型自身分布的训练数据，提升学习效率。然而，参考答案引入了强烈的风格偏差，导致生成模型模仿表面形式而非学习有用推理模式。我们发现重写数据中存在大量高困惑度（PPL）词元，其来源分为两类：有益的知识增强型逻辑修正，以及有害的由参考模仿引发的风格漂移。若对所有此类词元同等处理，会破坏基础模型的原始分布，尤其在困难推理任务上性能下降。为此，我们提出分布对齐自蒸馏（DASD），利用答案感知的参考模型生成候选词元，并根据基础模型的置信度动态过滤。DASD保留编码有用逻辑知识的词元，同时抑制分布不一致的风格噪声。在数学、代码和常识推理基准上的实验表明，DASD持续优于现有基线，减少高PPL词元，并提升不同难度任务下的鲁棒性。
原文摘要 · Abstract (English)
Self-distillation improves learning efficiency by rewriting reference answers as training data that better matches the model's own distribution. However, reference answers also introduce strong stylistic biases, causing the generative model to imitate surface forms rather than learn useful reasoning patterns. We observe that the rewriting data contains a large number of high-perplexity (PPL) tokens, coming from two distinct sources: beneficial knowledge-enhancing logical corrections, and harmful stylistic drift induced by reference imitation. Treating all such tokens equally can disrupt the base model's original distribution and degrade performance, especially on difficult reasoning tasks. To address this, we propose Distribution-Aligned Self-Distillation (DASD), which uses an answer-aware reference model to generate candidate tokens and dynamically filters them according to the base model's confidence. DASD preserves tokens that encode useful logical knowledge while suppressing distributionally misaligned style noise. Experiments on math, code, and commonsense reasoning benchmarks show that DASD consistently outperforms competitive baselines, reduces high-PPL tokens, and improves robustness across tasks of varying difficulty.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。