让小模型学会大模型的对话能力，效果更优且能超越老师。
daDPO: Distribution-Aware DPO for Distilling Conversational Abilities
- 基于教师模型输出分布优化小模型，提升对话质量。
- 20%剪枝的7B模型接近教师性能，误差仅-7.3%。
- 1.5B小模型偶尔胜过7B老师，适合资源受限场景。
大型语言模型在多种应用中表现卓越，但模型规模缩小后其对话能力急剧下降，制约了在资源受限环境中的部署。基于直接偏好优化的知识蒸馏（dDPO）成为提升小型模型对话能力的有前景方法。然而，现有方法多采用‘黑箱’蒸馏，仅利用教师模型的输出响应，忽视了其输出分布信息。本文提出daDPO（分布感知的DPO），一种统一的偏好优化与分布驱动蒸馏方法。通过严格的理论分析与实证验证，结果表明daDPO在恢复剪枝模型性能及增强小型LLM方面优于现有方法。特别是在领域内评估中，20%剪枝的Vicuna1.5-7B模型达到接近教师的表现（偏好率-7.3%，相较dDPO的-31%），而Qwen2.5-1.5B模型甚至偶尔超过其7B教师模型（胜率14.0%）。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated exceptional performance across various applications, but their conversational abilities decline sharply as model size decreases, presenting a barrier to their deployment in resource-constrained environments. Knowledge distillation with Direct Preference Optimization (dDPO) has emerged as a promising approach to enhancing the conversational abilities of smaller models using a larger teacher model. However, current methods primarily focus on 'black-box' KD, which only uses the teacher's responses, overlooking the output distribution offered by the teacher. This paper addresses this gap by introducing daDPO (Distribution-Aware DPO), a unified method for preference optimization and distribution-based distillation. We provide rigorous theoretical analysis and empirical validation, showing that daDPO outperforms existing methods in restoring performance for pruned models and enhancing smaller LLM models. Notably, in in-domain evaluation, our method enables a 20% pruned Vicuna1.5-7B to achieve near-teacher performance (-7.3% preference rate compared to that of dDPO's -31%), and allows Qwen2.5-1.5B to occasionally outperform its 7B teacher model (14.0% win rate).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。