拆解数据排序对大模型数学推理的影响,发现没有放之四海皆准的训练顺序。
What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoning
- 用五个维度量化题目难易,统一评估不同排序策略
- 正序与逆序效果取决于模型能力与任务复杂度,无绝对优劣
- 关注决策不确定性的样本能更好提升模型泛化能力
课程学习(CL)通过从易到难排序训练数据来提升大语言模型(LLMs)的推理能力,但现有研究使用不同的难度指标和训练设置,难以回答:课程学习何时有效?正序与逆序哪个更优?答案是否依赖评估指标?为此,本文提出一个统一的离线评估框架,将课程难度分解为五个互补维度:问题难度、模型意外度、置信度裕度、预测不确定性与决策变异性。在Llama3.1-8B、Mistral-7B和Gemma3-4B上对数学推理基准进行受控后训练实验,发现:(i) 不存在普适最优的课程策略——正序与逆序的效果取决于模型能力和任务复杂度;(ii) 即使在同一指标下,不同难度样本带来的增益也因任务需求而异;(iii) 任务对齐型课程聚焦于塑造模型最终表征与泛化能力,而内部状态型课程则调节置信度与不确定性等内部状态。研究挑战了‘通用课程策略’的假设,为不同模型与任务场景提供可操作指导,部分指标显示优先处理决策不确定样本可进一步提升学习效果。
原文摘要 · Abstract (English)
Curriculum learning (CL) - ordering training data from easy to hard - has become a popular strategy for improving reasoning in large language models (LLMs). Yet prior work employs disparate difficulty metrics and training setups, leaving open fundamental questions: When does curriculum help? Which direction - forward or reverse - is better? And does the answer depend on what we measure? We address these questions through a unified offline evaluation framework that decomposes curriculum difficulty into five complementary dimensions: Problem Difficulty, Model Surprisal, Confidence Margin, Predictive Uncertainty, and Decision Variability. Through controlled post-training experiments on mathematical reasoning benchmarks with Llama3.1-8B, Mistral-7B, and Gemma3-4B, we find that (i) no curriculum strategy dominates universally - the relative effectiveness of forward versus reverse CL depends jointly on model capability and task complexity; (ii) even within a single metric, samples at different difficulty levels produce distinct gains depending on task demands; and (iii) task-aligned curricula focus on shaping the model's final representations and generalization, whereas inner-state curricula modulate internal states such as confidence and uncertainty. Our findings challenge the notion of a universal curriculum strategy and offer actionable guidance across model and task regimes, with some metrics indicating that prioritizing decision-uncertain samples can further enhance learning outcomes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。