提出混合策略蒸馏方法，提升大模型压缩的稳定性和效率
Hybrid Policy Distillation for LLMs
- 将知识蒸馏统一为重加权对数似然目标，融合正反向KL散度优势
- 结合离线数据与轻量在线采样，在数学推理等任务上性能更优
- 适合需要高效压缩大模型的研究者和部署场景
知识蒸馏（KD）是压缩大语言模型（LLM）的有效方法，其效果取决于发散方向、优化策略和数据设置的复杂组合。本文剖析现有KD方法的设计，提出统一视角，将KD重新表述为基于标记级别的重加权对数似然目标。进一步提出混合策略蒸馏（HPD），融合正向与反向KL的优势，平衡模式覆盖与模式聚焦，并结合离线数据与轻量近似在线采样。在长序列数学推理及短文本对话、代码生成任务上验证了HPD，结果表明其在不同模型族和规模下均提升了优化稳定性、计算效率和最终性能。相关代码已开源。
原文摘要 · Abstract (English)
Knowledge distillation (KD) is a powerful paradigm for compressing large language models (LLMs), whose effectiveness depends on intertwined choices of divergence direction, optimization strategy, and data regime. We break down the design of existing KD methods and present a unified view that establishes connections between them, reformulating KD as a reweighted log-likelihood objective at the token level. We further propose Hybrid Policy Distillation (HPD), which integrates the complementary advantages of forward and reverse KL to balance mode coverage and mode-seeking, and combines off-policy data with lightweight, approximate on-policy sampling. We validate HPD on long-generation math reasoning as well as short-generation dialogue and code tasks, demonstrating improved optimization stability, computational efficiency, and final performance across diverse model families and scales. The code related to this work is available at https://github.com/zwhong714/Hybrid-Policy-Distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。