用合成数据提升符号回归外推性能,效果因数据集和模型而异。
Can Synthetic Data Improve Symbolic Regression Extrapolation Performance?
- 在数据稀疏区生成合成数据,通过知识蒸馏训练学生模型。
- 符号回归模型在外推区域性能显著提升,尤其用GP教师生成数据时。
- 适合需要可靠外推能力的科学建模与工程优化场景。
许多机器学习模型在训练数据范围内表现良好,但在外推时性能下降。使用遗传编程(GP)的符号回归(SR)虽灵活,但外推时不稳定。本文研究是否可通过添加合成数据改善这一问题。利用核密度估计(KDE)识别输入空间中数据稀疏区域,采用知识蒸馏方法生成合成数据:由教师模型在新输入点上预测,再用于训练学生模型。我们在六个基准数据集上评估该方法,以神经网络(NN)、随机森林(RF)和GP作为教师模型生成合成数据,并以它们为学生模型在增广数据上训练。结果表明,基于合成数据训练的GP模型在外推区域常有显著提升,尤其当使用GPe生成数据训练GPp时效果最佳;而在插值区域性能变化微小。还观察到输入空间内误差分布不均的现象。整体而言,该方法为改善外推提供了实用方案。注:此版本为2025年GECCO研讨会论文的修订版,修正了原稿多处内容。
原文摘要 · Abstract (English)
Many machine learning models perform well when making predictions within the training data range, but often struggle when required to extrapolate beyond it. Symbolic regression (SR) using genetic programming (GP) can generate flexible models but is prone to unreliable behaviour in extrapolation. This paper investigates whether adding synthetic data can help improve performance in such cases. We apply Kernel Density Estimation (KDE) to identify regions in the input space where the training data is sparse. Synthetic data is then generated in those regions using a knowledge distillation approach: a teacher model generates predictions on new input points, which are then used to train a student model. We evaluate this method across six benchmark datasets, using neural networks (NN), random forests (RF), and GP both as teacher models (to generate synthetic data) and as student models (trained on the augmented data). Results show that GP models can often improve when trained on synthetic data, especially in extrapolation areas. However, the improvement depends on the dataset and teacher model used. The most important improvements are observed when synthetic data from GPe is used to train GPp in extrapolation regions. Changes in interpolation areas show only slight changes. We also observe heterogeneous errors, where model performance varies across different regions of the input space. Overall, this approach offers a practical solution for better extrapolation. Note: An earlier version of this work appeared in the GECCO 2025 Workshop on Symbolic Regression. This arXiv version corrects several parts of the original submission.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。