通过识别训练中的关键瓶颈，动态选择学习令牌，提升大模型高效蒸馏效果。
Training-Trajectory-Aware Token Selection
- 基于训练轨迹动态筛选关键令牌，优化蒸馏路径
- 仅用数百样本，Qwen3-8B超越DeepSeek-R1，32B接近235B模型性能
- 适用于高阶推理模型的高效微调，尤其适合16B规模以下模型
高效蒸馏是将昂贵的推理能力转化为可部署效率的关键路径，但在学生模型已具备强推理能力的前沿场景中，简单的持续蒸馏常带来收益有限甚至性能下降。我们观察到一种典型训练现象：尽管损失单调下降，所有性能指标却在几乎同一瓶颈点突然骤降，随后缓慢恢复。进一步揭示了令牌层面的机制：置信度分化为两类——快速锚定优化的模仿锚点令牌与被抑制的待学令牌，二者无法共存是持续蒸馏失败的根本原因。为此，我们提出训练轨迹感知的令牌选择（T3S），在令牌层级重构训练目标，清除待学令牌的优化障碍。T3S在AR与dLLM设置中均实现稳定增益：仅需数百样本，Qwen3-8B即超越DeepSeek-R1，在竞争性推理基准上表现优异；Qwen3-32B接近Qwen3-235B；T3训练的LLaDA-2.0-Mini优于其AR基线，成为所有16B规模无思考模型中的最新最佳表现者。
原文摘要 · Abstract (English)
Efficient distillation is a key pathway for converting expensive reasoning capability into deployable efficiency, yet in the frontier regime where the student already has strong reasoning ability, naive continual distillation often yields limited gains or even degradation. We observe a characteristic training phenomenon: even as loss decreases monotonically, all performance metrics can drop sharply at almost the same bottleneck, before gradually recovering. We further uncover a token-level mechanism: confidence bifurcates into steadily increasing Imitation-Anchor Tokens that quickly anchor optimization and other yet-to-learn tokens whose confidence is suppressed until after the bottleneck. And the characteristic that these two types of tokens cannot coexist is the root cause of the failure in continual distillation. To this end, we propose Training-Trajectory-Aware Token Selection (T3S) to reconstruct the training objective at the token level, clearing the optimization path for yet-to-learn tokens. T3S yields consistent gains in both AR and dLLM settings: with only hundreds of examples, Qwen3-8B surpasses DeepSeek-R1 on competitive reasoning benchmarks, Qwen3-32B approaches Qwen3-235B, and T3-trained LLaDA-2.0-Mini exceeds its AR baseline, achieving state-of-the-art performance among all of 16B-scale no-think models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。