首次揭示符号回归的计算量缩放规律,可预测模型性能。
Towards Scaling Laws for Symbolic Regression
- 用端到端Transformer和可控数据生成研究缩放规律
- 验证损失与求解率均随算力呈幂律下降,跨越三个数量级
- 发现最优超参数与模型规模正相关,令牌/参数比约15
符号回归(SR)旨在发现解释观测数据的底层数学表达式,具有提供科学洞见和生成可解释、泛化能力强的表格数据模型的潜力。本文聚焦于SR的基础问题。基于深度学习的SR方法近年来已与遗传编程相媲美,但规模的作用仍缺乏系统研究。受语言建模缩放定律启发,我们首次系统性地探究了SR中的缩放现象,采用可扩展的端到端Transformer流水线和精心生成的训练数据。在五个不同模型规模、跨越三个数量级算力的实验中,我们发现验证损失和求解率均随算力呈现清晰的幂律趋势。进一步识别出计算最优的超参数缩放关系:最优批次大小和学习率随模型规模增长,且在我们的实验条件下,令牌/参数比约为15时表现最佳,随着算力增加略有上升。这些结果表明SR性能可由算力大致预测,并为下一代SR模型的训练提供了重要指导。
原文摘要 · Abstract (English)
Symbolic regression (SR) aims to discover the underlying mathematical expressions that explain observed data. This holds promise for both gaining scientific insight and for producing inherently interpretable and generalizable models for tabular data. In this work we focus on the basics of SR. Deep learning-based SR has recently become competitive with genetic programming approaches, but the role of scale has remained largely unexplored. Inspired by scaling laws in language modeling, we present the first systematic investigation of scaling in SR, using a scalable end-to-end transformer pipeline and carefully generated training data. Across five different model sizes and spanning three orders of magnitude in compute, we find that both validation loss and solved rate follow clear power-law trends with compute. We further identify compute-optimal hyperparameter scaling: optimal batch size and learning rate grow with model size, and a token-to-parameter ratio of $\approx$15 is optimal in our regime, with a slight upward trend as compute increases. These results demonstrate that SR performance is largely predictable from compute and offer important insights for training the next generation of SR models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。