重新审视思维链蒸馏中的能力差距，发现旧评估方法掩盖了性能下降问题。
Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective
- 提出更真实的评估协议，避免仅比较蒸馏后结果
- 发现教师与学生能力差距不总导致性能下降
- 为选择合适的师生模型对提供实用指导
思维链（CoT）蒸馏将强教师模型的推理行为迁移到小型学生模型，但以往研究指出：当教师与学生能力差异过大时，蒸馏可能失败。本文从实用角度重新审视该能力差距问题，重新考察常用实验设置。值得注意的是，我们发现CoT蒸馏常导致性能低于学生模型的预蒸馏基线，这一问题在仅报告蒸馏后对比时被掩盖。因此，我们提出了更合理的评估协议，并发现能力差距的影响在不同任务和设置中并不始终主导，尤其当候选教师性能差异显著时。研究结果为实际选择教师-学生配对提供了重要参考。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) distillation transfers reasoning behaviors from a strong teacher to a smaller student, but prior work reports a capacity gap: distillation may fail when the teacher-student capability mismatch is large. We revisit the capacity gap from a practical perspective by re-examining commonly used experimental settings. Notably, we find that CoT distillation often degrades performance compared to the student's pre-distillation baseline, an issue obscured when only post-distillation comparisons are reported. We therefore propose a more realistic evaluation protocol and find that the impact of capacity gap effects does not consistently dominate across tasks and settings, especially when candidate teachers differ substantially in performance. Our results offer practical guidance for selecting teacher-student pairs in CoT distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。