解决大模型推理蒸馏中的分布偏移问题,提升小模型推理准确性。
Distribution Corrected Offline Data Distillation for Large Language Models

- 通过自适应强调与学生推理分布更匹配的教师监督信号
- 在GSM8K等数学推理任务上准确率显著优于现有离线蒸馏方法
- 无需在线采样,适合资源受限场景下的高效推理模型训练
将强大学习模型的推理轨迹蒸馏到小型模型中,是资源受限环境下提升智能的有效途径。现有方法面临根本性权衡:离线蒸馏虽提供高质量、样本高效的监督,但存在分布漂移问题——训练时学生依赖教师生成的前缀,推理时则自回归生成前缀,导致长推理链中误差累积。而在线或自蒸馏方法虽更贴近学生推理分布,却需昂贵的在线采样,且早期训练生成的轨迹质量较低。本文提出一种原则性的离线推理蒸馏框架,在保留离线教师数据高效性和监督质量的同时,纠正教师-学生分布漂移。该方法自适应强化与学生在线策略分布更一致的教师监督。在GSM8K、MATH、MATH500及更难的竞赛类任务(AMC、AIME、OlympiadBench)上的评估显示,本方法在推理准确率上超越先前离线蒸馏算法,同时生成更稳定的推理轨迹,并保持指令遵循能力。结果表明,轻量级、具备分布校正意识的训练可显著增强离线推理蒸馏,无需在线滚动生成。
原文摘要 · Abstract (English)
Distilling reasoning traces from strong large language models into smaller ones is a promising route to improve intelligence in resource-constrained settings. Existing approaches face a fundamental trade-off: offline distillation from teacher-generated traces provides high-quality, sample-efficient supervision but suffers from distributional drift: during training, the student model conditions on teacher-generated prefixes, whereas during inference the student autoregresses on self-generated prefixes, leading to compounding errors over long reasoning trajectories. Meanwhile, on-policy or self-distillation methods better match the student's inference-time distribution, but require costly online sampling and often produce low-quality traces in early training. We propose a principled offline reasoning distillation framework that preserves the efficiency and supervision quality of offline teacher-generated data while correcting teacher-student distribution drift. It adaptively emphasizes teacher supervision that is better aligned with the student's on-policy distribution. Evaluations on mathematical reasoning benchmarks of GSM8K, MATH, MATH500, and harder held-out competition-style tasks, including AMC, AIME, and OlympiadBench, show that our method improves reasoning accuracy over prior offline distillation algorithms and yields more stable reasoning traces while preserving instruction-following capabilities. Our work shows that lightweight, distribution-correction-aware training can substantially strengthen offline reasoning distillation without online rollouts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。