用神经网络预测模型在不同规模下的表现，比传统方法更准。
Neural Neural Scaling Laws
- 将性能预测建模为时间序列外推，融合训练轨迹与损失数据。
- 在66个任务上误差仅1.99%，比传统方法降低44%。
- 无需预设函数形式，能泛化到未见的模型和任务。
神经缩放定律可预测语言模型性能随训练数据增加而提升的趋势。尽管整体验证损失可能呈现平滑的幂律曲线，但具体下游任务的表现各异：部分持续提升，部分趋于饱和，甚至有些随规模增大而下降。现有方法依赖验证损失预测下游性能存在两大缺陷：逐标记损失平均会掩盖有效信号，且无法用单一参数形式覆盖所有缩放行为。为此，我们提出神经神经缩放定律（NeuNeu），一个将缩放预测视为时间序列外推的神经网络模型。NeuNeu结合观测到的准确率轨迹与标记级验证损失，学习直接预测未来性能，避免了对特定函数形式的假设。模型完全基于HuggingFace上的开源模型检查点训练，在66个下游任务上实现1.99%的平均绝对误差（MAE），相比逻辑缩放定律（3.56% MAE）降低44%。此外，NeuNeu具备零样本泛化能力，适用于未见过的模型家族、架构、参数量及下游任务。结果表明，直接从数据中学习缩放规律优于传统参数化方法。
原文摘要 · Abstract (English)
Neural scaling laws predict how language model performance improves with increased training inputs. While aggregate metrics like validation loss can follow smooth power-law curves, individual downstream tasks exhibit diverse scaling behaviors: some improve monotonically, others plateau, and some even degrade with scale. We argue that predicting downstream performance from validation loss suffers from two limitations: averaging token-level losses obscures signal, and no simple parametric family can capture the full spectrum of scaling behaviors. To address this, we propose Neural Neural Scaling Laws (NeuNeu), a neural network that frames scaling law prediction as time-series extrapolation. NeuNeu combines temporal context from observed accuracy trajectories with token-level validation losses, learning to predict future performance without the limitations inherent in assuming a specific functional form. Trained entirely on open-source model checkpoints from HuggingFace, NeuNeu achieves 1.99% mean absolute error in predicting model accuracy on 66 downstream tasks -- a 44% reduction compared to logistic scaling laws (3.56% MAE). Furthermore, NeuNeu generalizes zero-shot to unseen model families, architectures, parameter counts, and downstream tasks. Our work suggests that predicting downstream scaling directly from data outperforms parametric alternatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。