用新指标选更有效的推理路径,让小模型学得更好
Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment
- 提出RSR指标,同时衡量推理路径与学生模型的匹配度和信息量
- 在5个学生模型上验证,RSR与最终推理表现相关性达0.86
- 适合需要高效知识蒸馏的模型训练者使用
长链推理(CoT)轨迹为将推理能力从教师模型蒸馏到学生大模型提供了丰富的监督信号。然而,先前研究及我们的实验表明,更强教师模型生成的轨迹并不一定带来更好的学生模型,凸显了数据-学生适配性的重要性。现有方法主要通过学生模型的似然度评估适配性,倾向于选择与学生当前行为高度一致的轨迹,却忽略了更具信息量的路径。为此,我们提出秩惊喜比(RSR),一个简洁的指标,同时捕捉轨迹的对齐性与信息量。其核心思想是:有效轨迹通常在学生模型中表现为低绝对概率但相对高排名的词元。具体地,RSR定义为轨迹平均词元秩与平均负对数似然的比值,计算简便且易于解释。在5个学生模型和11个不同教师模型生成的推理轨迹上,RSR与蒸馏后推理性能的平均斯皮尔曼相关系数达0.86,显著优于现有指标。我们进一步验证了其在轨迹筛选与教师选择中的实际效用。
原文摘要 · Abstract (English)
Long chain-of-thought (CoT) trajectories provide rich supervision signals for distilling reasoning from teacher to student LLMs. However, both prior work and our experiments show that trajectories from stronger teachers do not necessarily yield better students, highlighting the importance of data-student suitability in distillation. Existing methods assess suitability primarily through student likelihood, favoring trajectories that align closely with the student model's current behavior but overlooking more informative ones. Addressing this, we propose Rank-Surprisal Ratio (RSR), a simple metric that captures both alignment and informativeness to assess the suitability of a reasoning trajectory. RSR is motivated by the observation that effective trajectories typically balance learning signal strength and behavioral alignment by combining low absolute probability with relatively high-ranked tokens under the student model. Concretely, RSR is defined as the ratio of a trajectory's average token-wise rank to its average negative log-likelihood, and is straightforward to compute and interpret. Across five student models and reasoning trajectories from 11 diverse teachers, RSR strongly correlates with post-training reasoning performance (average Spearman 0.86), consistently outperforming existing metrics. We further demonstrate its practical utility in both trajectory selection and teacher selection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。