选答案不看老师强弱，而要看对学生的帮助大小。
When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection
- 根据学生当前水平选择最适合作业答案，而非选最强老师
- 在6个任务上提升30个学生模型表现，最高增益达12.7%
- 适合想提升模型蒸馏效果的研究者和工程师
大模型训练越来越依赖教师生成的监督信号，包括合成回答、推理过程和工具使用示范。当前做法通常选择性能最强的教师生成学生训练数据，隐含假设是教师测试成绩能代表教学能力。我们发现这一假设可能失效：即使多个教师对同一问题给出正确答案，最强教师的答案未必最适合某个特定学生。为此，我们提出学生中心答案采样（SCAS）框架，从已验证的教师答案中，根据其对学生学习成本的估计进行选择。基于逐标记梯度分解，我们推导出一种仅需前向传播的高效代理指标，并用于训练中的答案选择。在30个教师模型、6个学生基础模型和6个任务上的实验表明，SCAS持续提升学生表现，说明有效知识蒸馏应优先考虑与当前学生匹配的监督，而非仅看教师强度。
原文摘要 · Abstract (English)
LLM training increasingly relies on teacher-generated supervision, from synthetic responses to reasoning traces and tool-use demonstrations. Current practice often chooses the highest-performing teacher to generate student training data, implicitly treating teacher test performance as a proxy for teaching quality. We show that this assumption can fail: even when multiple teachers provide correct answers to the same question, the answer from the strongest teacher is not necessarily the best supervision for a given student. To address this gap, we propose Student-Centric Answer Sampling (SCAS), a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost. Motivated by a token-wise gradient decomposition, we derive an efficient forward-only proxy for this cost and use it to guide answer selection during training. Experiments across 30 teacher models, 6 student base models, and 6 tasks show that SCAS consistently improves student performance, suggesting that effective distillation should prioritize supervision matched to the current student rather than teacher strength alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。