模型越大,越适合用低概率数据训练,小模型则需高概率数据。
Capacity-Dependent Effects of Data Selection for Reasoning

- 按模型大小调整数据选择:小模型用高似然数据,大模型用低似然数据
- 80亿参数模型训练更久后,低似然数据提升效果比高似然更好
- 揭示了模型容量与数据难度的协同作用,指导实际训练策略
在推理型监督微调中,同一指令的不同候选回复与学生模型当前分布的匹配度差异显著。近期基于似然的数据选择方法认为,更接近学生分布的回复能提供更有效的监督,因而假设高似然回复通常更优。本文重新审视这一直觉,发现其有效性高度依赖于模型容量和训练时长。我们在数学推理任务上进行受控实验,使用1.5B至8B参数的学生模型,由更强教师模型生成监督信号。结果呈现明显的“快适配/慢增益”模式:高似然数据在早期加速小模型收敛,但当训练时间延长,80亿参数模型在低似然数据下表现更优。通过学习动态分析发现,小模型难以吸收低似然监督,易陷入浅层或重复行为;而大模型能有效向教师分布迁移。我们进一步提出容量受限的蒸馏理论框架,阐明数据难度、数据跨度与学生容量共同决定知识传递效率。结论表明,推理任务的数据选择应结合模型容量与计算预算,而非一味偏好高似然数据。
原文摘要 · Abstract (English)
In reasoning supervised fine-tuning, candidate responses for the same instruction can differ substantially in how well they match the student's current distribution. Recent likelihood-based response selection methods suggest that responses closer to the student distribution provide more effective supervision, motivating the hypothesis that high-likelihood responses may generally be preferable for fine-tuning. In this paper, we revisit this intuition and show that the value of likelihood-based data selection depends critically on model capacity and training duration. Through controlled experiments on mathematical reasoning, using students ranging from 1.5B to 8B parameters and supervision generated by stronger teacher models, we observe a clear \emph{capacity-dependent} ``{\color{SMALLCOLOR}\textbf{Fast-Fit}} / {\color{LARGECOLOR}\textbf{Slow-Gain}}'' pattern. High-likelihood data provides faster and more stable early improvements, especially for smaller models, but low-likelihood data becomes increasingly beneficial for larger models when training is allowed to continue longer. To explain this phenomenon, we analyze learning dynamics, showing that small models often fail to absorb low-likelihood supervision and instead fall into shallow or repetitive behaviors, while larger models are better able to move toward the teacher distribution under such data. We further provide a capacity-constrained theoretical view of distillation that clarifies how data difficulty, data span, and student capacity jointly govern transfer. Overall, our findings show that effective data selection for reasoning should be aware of model capacity and computing budget rather than based on a single universal preference for high-likelihood supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。