AdaSwitch动态切换教学模式，让小模型更好学大模型。
AdaSwitch: Balancing Exploration and Guidance in Knowledge Distillation via Adaptive Switching
- 根据学生能力自动切换自产和教师指导的生成方式
- 在三个数据集上准确率提升，推理能力增强
- 适合资源受限下追求性能的小模型训练
小型语言模型（SLMs）在延迟与计算资源严格受限的应用中至关重要，但实现高性能仍具挑战。知识蒸馏（KD）可将大模型能力迁移至小模型，但现有方法面临两难：离策略蒸馏虽提供高质量监督，却存在暴露偏差（训练与推理不一致）；同策略方法虽保证一致性，但受限于学生生成输出质量低。为此，我们提出AdaSwitch，一种通过自适应切换机制动态结合同策略与离策略生成的新方法。AdaSwitch允许学生在其能力范围内自主探索预测，并仅在偏差超过上下文感知阈值时选择性引入教师指导。该范式在保持生成一致性的同时确保高质量监督。在三个数据集上的实验表明，AdaSwitch在适度开销下持续提升准确率与推理能力。
原文摘要 · Abstract (English)
Small language models (SLMs) are crucial for applications with strict latency and computational constraints, yet achieving high performance remains challenging. Knowledge distillation (KD) can transfer capabilities from large teacher models, but existing methods face a dilemma: off-policy distillation provides high-quality supervision but suffers from exposure bias (training inference mismatch), while on-policy approaches ensure consistency but are limited by the low quality of student-generated outputs. To address these issues, we propose AdaSwitch, a novel approach that dynamically combines on-policy and off-policy generation via an adaptive switching mechanism. AdaSwitch allows the student to explore its predictions within its capability and selectively integrates teacher guidance only when divergence exceeds a context-aware threshold. This paradigm preserves generation consistency while ensuring high-quality supervision. Experiments on three datasets demonstrate that AdaSwitch consistently improves accuracy and reasoning capability with moderate overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。