arXiv:2604.18963cs.LGcs.AI2026-04ACL

通过强化微调调控大模型可蒸馏性,提升小模型性能并保护模型知识产权。

Distillation Traps and Guards: A Calibration Knob for LLM Distillability

论文配图:Distillation Traps and Guards: A Calibration Knob for LLM Distillability
图 1 · 摘自论文原文
  • 用强化微调后处理校准大模型的可蒸馏性,控制知识转移效果。
  • 校准后教师蒸馏出的学生在数学、问答等任务上超越基准模型。
  • 既能提升蒸馏效果,又可防止不可蒸馏模型的知识泄露,适合部署安全场景。

知识蒸馏(KD)将大语言模型(LLM)的能力迁移至小型学生模型,但常因不可预测地失败且存在模型泄露风险。我们分析发现三类蒸馏陷阱:尾部噪声、非策略不稳定性及师生差距,导致训练信号扭曲,表现为过度自信的幻觉、自我修正崩溃和局部解码退化。基于此,我们提出一种后处理校准方法,据我们所知首次实现通过强化微调(RFT)对教师模型的可蒸馏性进行可控调节。目标函数结合任务效用、KL锚点与跨分词器校准奖励,使可蒸馏性成为基础模型的实际安全控制开关,连接稳健的师生迁移与部署感知的模型防护。在数学、知识问答和指令跟随任务上的实验表明,从可蒸馏校准教师中蒸馏的学生模型优于SFT与传统KD基线;而不可蒸馏校准教师虽保持自身任务性能,却导致学生模型崩溃,为高效蒸馏与模型知识产权保护提供实用控制钮。

原文摘要 · Abstract (English)

Knowledge distillation (KD) transfers capabilities from large language models (LLMs) to smaller students, yet it can fail unpredictably and also underpins model leakage risks. Our analysis revealed several distillation traps: tail noise, off-policy instability, and, most fundamentally, the teacher-student gap, that distort training signals. These traps manifest as overconfident hallucinations, self-correction collapse, and local decoding degradation, causing distillation to fail. Motivated by these findings, we propose a post-hoc calibration method that, to the best of our knowledge, for the first time enables control over a teacher's distillability via reinforcement fine-tuning (RFT). Our objective combines task utility, KL anchor, and across-tokenizer calibration reward. This makes distillability a practical safety lever for foundation models, connecting robust teacher-student transfer with deployment-aware model protection. Experiments across math, knowledge QA, and instruction-following tasks show that students distilled from distillable calibrated teachers outperform SFT and KD baselines, while undistillable calibrated teachers retain their task performance but cause distilled students to collapse, offering a practical knob for both better KD and model IP protection.

知识蒸馏大模型安全强化微调模型保护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。