大规模实证发现经典机器学习误差随数据量呈幂律下降，且同模型类有近似规律。
Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study
- 127名学生在18个数据集上运行3种模型，共完成1.15万次训练并拟合幂律曲线。
- 树模型在全量数据下表现最优，77.7%的曲线幂律拟合优度超过0.8。
- 模型家族内部指数近似统一，但实现细节差异导致指数波动达0.144。
以往经典机器学习的学习曲线研究多在小规模下进行，通常仅限单一曲线、单个团队、少量实验。本文开展分布式课堂级复现：127名研究生各自在3个指定数据集上执行固定流程，覆盖18个表格分类与回归数据集及6类模型（提升、随机森林、SVM、线性/逻辑回归、岭回归、Lasso），共产生11,536次训练和1,648条误差随样本数变化的幂律曲线（error(N) = a N^(-b) + c）。主要发现：(1) 幂律拟合效果良好，77.7%的实验单元R² > 0.8，树模型在全量数据下占优（提升算法覆盖50%数据集，随机森林33%）；线性模型在分类任务中表现较差。(2) 同一模型家族内指数近似一致：5/6模型族用单一家庭级指数预测跨数据集曲线，性能几乎等同于逐数据集拟合（R²差距<0.011），但曲线压缩不完全（32–58%点落在±0.5 dex内），表明为近似可压缩性，非数据无关普适性；Lasso为负控失败，岭回归在留一数据集测试中敏感。(3) 实现差异显著：即使固定random_state=42，不同实现间指数均值偏差达CV(b)=0.144，源于协议中未约束部分（预处理、编码、缺失值处理）所致，非种子差异。论文发布聚合曲线、每单元拟合结果及达到目标误差0.15所需数据量表。
原文摘要 · Abstract (English)
Prior classical-ML learning-curve work fits power laws to tree, linear, and kernel models on tabular data, but at small scale: typically one curve, one team, a handful of cells. We present a distributed classroom-scale replication: 127 graduate students each ran a fixed protocol on 3 assigned datasets, drawn from 18 tabular classification and regression datasets and 6 model families (Boosting, Random Forest, SVM, Linear/Logistic, Ridge, Lasso), yielding 11,536 training runs and 1,648 fitted power-law curves of the form error(N) = a N^(-b) + c. Three findings. (1) Power laws fit: R^2 > 0.8 on 77.7% of cells, with tree ensembles dominating at full data (Boosting 50% of datasets, RandomForest 33%; linear models underperform on classification). (2) Approximate shared exponents within a model family: for 5 of 6 families, a single family-level exponent predicts each family's cross-dataset curves nearly as well as per-dataset exponents (R^2 gap < 0.011), though AIC favors the unconstrained fit and curve collapse is partial (32-58% of points within +/-0.5 dex). We frame this as approximate predictive compressibility, not dataset-independent universality; Lasso fails outright (negative control) and Ridge is fragile under leave-one-dataset-out. (3) Replicator-implementation variance: with random_state=42 fixed, independent re-implementations of the same protocol still differ by mean CV(b) = 0.144 on the fitted exponent -- not seed variance, but the spread induced by unconstrained parts of the protocol (preprocessing, encoding, missing-value handling). We release the aggregated curves, per-cell fits, and a practical data-requirement table for N* to reach target error 0.15.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。