arXiv:2604.03873cs.LGcs.CL2026-04被引 4

用静态学生输出对比教师最优响应,实现高效高质大模型蒸馏。

SODA: Semi On-Policy Black-Box Distillation for Large Language Models

  • 用教师最优输出与学生静态输出配对构造对比信号,避免动态生成和对抗训练。
  • 在16项基准测试中15项超越或持平当前最佳,训练速度提升10倍。
  • 适合追求高效、稳定蒸馏的模型压缩场景,尤其适用于资源受限部署。

大语言模型的黑盒知识蒸馏面临严格权衡:简单离策略方法难以纠正学生模型固有错误,全在线策略方法虽能解决但引入训练不稳定性与高昂计算开销。为此,我们提出SODA(基于对齐的半在线蒸馏),利用前沿教师模型与小型基础模型之间的能力差距。由于紧凑学生模型的零样本输出几乎始终劣于教师目标,仅需将教师最优响应与一次静态的学生输出快照配对,即可构建高效对比信号。实验证明,让小模型暴露于自身静态低质行为已足够实现高质量分布对齐,无需昂贵动态采样与脆弱对抗平衡。在四个紧凑版Qwen2.5与Llama-3模型上评估显示,SODA在16项基准中的15项超越或持平当前最佳表现,同时训练速度提升10倍,峰值显存减少27%,彻底消除对抗不稳定性。

原文摘要 · Abstract (English)

Black-box knowledge distillation for large language models presents a strict trade-off. Simple off-policy methods (e.g., sequence-level knowledge distillation) struggle to correct the student's inherent errors. Fully on-policy methods (e.g., Generative Adversarial Distillation) solve this via adversarial training but introduce well-known training instability and crippling computational overhead. To address this dilemma, we propose SODA (Semi On-policy Distillation with Alignment), a highly efficient alternative motivated by the inherent capability gap between frontier teachers and much smaller base models. Because a compact student model's natural, zero-shot responses are almost strictly inferior to the powerful teacher's targets, we can construct a highly effective contrastive signal simply by pairing the teacher's optimal response with a one-time static snapshot of the student's outputs. This demonstrates that exposing the small student to its own static inferior behaviors is sufficient for high-quality distribution alignment, eliminating the need for costly dynamic rollouts and fragile adversarial balancing. Extensive evaluations across four compact Qwen2.5 and Llama-3 models validate this semi on-policy paradigm. SODA matches or outperforms the state-of-the-art methods on 15 out of 16 benchmark results. More importantly, it achieves this superior distillation quality while training 10 times faster, consuming 27% less peak GPU memory, and completely eliminating adversarial instability.

知识蒸馏大模型压缩高效训练分布对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。