提出对抗性蒸馏评估框架，揭示现有防御的脆弱性。
The Distillation Game: Adaptive Attacks & Efficient Defenses
- 构建师生博弈模型，教师在有限效用下防御蒸馏攻击。
- 发现被动评估低估攻击能力，自适应学生可恢复更多性能。
- 提出PoE防御方法，低成本且保留高质量推理过程。
蒸馏攻击使模型提供者面临部署权衡：提升模型实用性的输出也更容易被模仿。本文通过构建效用受限的教师与自适应学生之间的极小极大博弈，研究这一权衡。框架导出可计算的一侧响应规则：学生侧采用重加权高价值样本的自适应评估策略，教师侧则设计抑制蒸馏有用输出的防御模板。基于一个廉价示例价值代理，我们提出Product-of-Experts（PoE）——一种仅需前向传播的防御机制，将教师与代理学生结合生成结果。实验证明，自适应评估揭示显著的被动-自适应差距：在GSM8K和MATH数据集上，自适应学生比被动评估预测恢复了更多能力。在此更强评估下，昂贵防御与PoE之间的鲁棒性差距大幅缩小，而PoE仍保持更低开销并保留更高质量的推理轨迹。结果表明，强蒸馏仍难彻底遏制，反蒸馏进展应以自适应学生为基准衡量。代码已公开：https://github.com/ysfalh/distillation-game。
原文摘要 · Abstract (English)
Distillation attacks create a deployment trade-off for model providers: the same outputs that make a model more useful can also make it easier to imitate. We study this trade-off through a minimax game between a utility-constrained teacher and an adaptive student. Our framework yields tractable one-sided response rules: an adaptive evaluation rule in which the student reweights high-value examples, and a teacher-side defense template that suppresses outputs most useful for distillation. From a cheap proxy for example value, we derive Product-of-Experts (PoE), a simple forward-pass-only defense that combines the teacher with a proxy student during generation. Empirically, adaptive evaluation reveals a large passive--adaptive gap: on state-of-the-art defenses, adaptive students recover substantially more capability than passive evaluation suggests on GSM8K and MATH. Under this stronger evaluation, the apparent robustness gap between expensive defenses and PoE narrows considerably, while PoE remains substantially cheaper and preserves higher-quality reasoning traces. Overall, our results suggest that strong distillation remains difficult to stop, and that progress on antidistillation should be judged against adaptive students rather than passive ones. Our code is available at: https://github.com/ysfalh/distillation-game.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。