揭示学习曲线比想象中更不规则，挑战传统假设。
LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thought
- 构建高分辨率学习曲线数据库LCDB 1.1，覆盖多种现代模型。
- 约15%学习曲线存在显著不规则，是此前估计的两倍。
- 发现特定模型更易出现不规则，对模型选择有实际影响。
样本级学习曲线描绘性能随训练集大小的变化，有助于研究规模定律、加速超参数调优和模型选择。以往常假设学习曲线是良好行为的：单调递增且凸性。通过构建大规模的Learning Curves Database 1.1（LCDB 1.1），包含高分辨率学习曲线及更多现代学习器（CatBoost、TabNet、RealMLP和TabPFN），我们发现学习曲线的实际表现远不如预期规整。采用严格的统计方法，观察到约15%的学习曲线存在显著不规则，几乎为先前估计的两倍。我们还识别出具体导致不规则的模型，并证明不同特征缩放方法极少能缓解此问题。评估其对下游任务（如学习曲线拟合和模型选择）的影响后发现，不规则行为带来重大挑战，凸显了LCDB 1.1作为未来研究严苛基准的重要性。
原文摘要 · Abstract (English)
Sample-wise learning curves plot performance versus training set size. They are useful for studying scaling laws and speeding up hyperparameter tuning and model selection. Learning curves are often assumed to be well-behaved: monotone (i.e. improving with more data) and convex. By constructing the Learning Curves Database 1.1 (LCDB 1.1), a large-scale database with high-resolution learning curves including more modern learners (CatBoost, TabNet, RealMLP and TabPFN), we show that learning curves are less often well-behaved than previously thought. Using statistically rigorous methods, we observe significant ill-behavior in approximately 15% of the learning curves, almost twice as much as in previous estimates. We also identify which learners are to blame and show that specific learners are more ill-behaved than others. Additionally, we demonstrate that different feature scalings rarely resolve ill-behavior. We evaluate the impact of ill-behavior on downstream tasks, such as learning curve fitting and model selection, and find it poses significant challenges, underscoring the relevance and potential of LCDB 1.1 as a challenging benchmark for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。