让大模型教小模型时，只重点学可信的词，提升压缩效果。
SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs
- 按教师信心筛选关键词进行教学，拒绝低信噪比内容。
- 在多种任务上超越现有方法，小模型性能达顶尖水平。
- 无需改结构或额外模型，适配各种训练数据和场景。
知识蒸馏是将大语言模型压缩为小型学生模型的标准方法，但多数流程对所有词统一施加损失，忽略教师置信度。这种无差别监督会放大噪声与高熵信号，在教师-学生容量差异较大时尤为有害。本文提出SelecTKD，一种即插即用的有选择性令牌加权蒸馏框架，将关注点从‘如何衡量差异’转向‘何处施加学习’。每一步中，学生提出候选词，由教师通过稳健的‘提议-验证’机制判断：采用贪心Top-k或非贪心Spec-k两种方式。被接受的词获得完整损失，被拒绝的词则被掩码或降权。该目标无关设计适用于在线与离线策略数据，诱导出可量化的隐式课程（令牌接受率，TAR），并稳定优化过程。在指令遵循、数学推理、代码生成及视觉语言模型设置下，SelecTKD持续优于强基线，在不改变模型架构或引入额外参考模型的前提下，实现了小模型的最先进性能。
原文摘要 · Abstract (English)
Knowledge distillation (KD) is a standard route to compress Large Language Models (LLMs) into compact students, yet most pipelines uniformly apply token-wise loss regardless of teacher confidence. This indiscriminate supervision amplifies noisy, high-entropy signals and is especially harmful under large teacher-student capacity gaps. We introduce SelecTKD, a plug-and-play Selective Token-Weighted distillation framework that shifts the focus from "how to measure divergence" to "where to apply learning". At each step, the student proposes tokens that are verified by the teacher through a robust propose-and-verify procedure with two variants: greedy Top-k and non-greedy Spec-k. Accepted tokens receive full loss, while rejected tokens are masked or down-weighted. This objective-agnostic design works with on- and off-policy data, induces an implicit curriculum quantified by Token Acceptance Rate (TAR), and stabilizes optimization. Across instruction following, mathematical reasoning, code generation, and a VLM setting, SelecTKD consistently improves strong baselines and achieves state-of-the-art results for small models without architectural changes or extra reference models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。