预训练符号回归模型在分布内表现好，但跨分布泛化能力差。
Analyzing Generalization in Pre-Trained Symbolic Regression
- 用大规模预训练+Transformer结构替代传统组合搜索
- 分布外任务性能显著下降，存在明显泛化鸿沟
- 适合关注模型可靠性与实际应用的科研人员
符号回归算法在数学表达式空间中寻找能解释给定数据的公式。基于Transformer的模型通过将昂贵的组合搜索转移到大规模预训练阶段，成为一种有前景且可扩展的方法。然而，这些模型的成功高度依赖预训练数据。其对预训练分布之外问题的泛化能力尚未被充分探索。本文系统评估了预训练、基于Transformer的符号回归模型的泛化能力，对多个前沿方法在预训练分布内及一系列分布外挑战上的表现进行了严格测试。结果揭示出显著的两极分化：预训练模型在分布内表现良好，但在分布外场景下性能持续下降。我们得出结论，这一泛化差距是实践中的关键障碍，严重限制了预训练方法在真实世界应用中的可用性。
原文摘要 · Abstract (English)
Symbolic regression algorithms search a space of mathematical expressions for formulas that explain given data. Transformer-based models have emerged as a promising, scalable approach shifting the expensive combinatorial search to a large-scale pre-training phase. However, the success of these models is critically dependent on their pre-training data. Their ability to generalize to problems outside of this pre-training distribution remains largely unexplored. In this work, we conduct a systematic empirical study to evaluate the generalization capabilities of pre-trained, transformer-based symbolic regression. We rigorously test performance both within the pre-training distribution and on a series of out-of-distribution challenges for several state of the art approaches. Our findings reveal a significant dichotomy: while pre-trained models perform well in-distribution, the performance consistently degrades in out-of-distribution scenarios. We conclude that this generalization gap is a critical barrier for practitioners, as it severely limits the practical use of pre-trained approaches for real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。