用合成数据让基础模型学会工程回归,无需真实数据训练。
Engineering Regression Without Real-Data Training: Domain Adaptation for Tabular Foundation Models Using Multi-Dataset Embeddings
- 基于多数据集嵌入识别工程数据特征,生成类工程的合成数据。
- 仅用合成数据微调后,在35个工程数据集上平均提升4.44倍数据效率。
- 适合数据稀缺的工业与科研场景,突破真实数据获取瓶颈。
工程预测建模长期依赖定制模型和小规模孤立的表格数据,限制了大规模学习的应用。尽管表格基础模型取得进展,但预训练所用的合成数据分布可能不反映工程数据的统计特性,影响迁移效果。我们提出TREDBench,一个包含83个真实世界表格回归数据集的标注集合,其中包含专家标记的工程/非工程类别。利用TabPFN 2.5的样本级嵌入,在统一表示空间中分析领域结构,发现工程数据部分可区分于非工程数据,而标准程序生成数据则与工程数据显著不同,揭示了显著的合成-真实领域差距。为在不使用真实工程样本的前提下弥合这一差距,我们提出一种嵌入引导的合成数据筛选方法:生成并识别‘类工程’合成数据集,仅用这些选定任务对TabPFN 2.5进行持续预训练。在35个工程回归数据集上,该方法在预测准确率和数据效率方面均优于初始模型(29/35胜出)及AutoGluon(27/35胜出),平均乘法数据效率提升分别为1.75倍和4.44倍。结果表明,有原则地构建合成数据可将程序生成器转化为领域相关的‘数据引擎’,使基础模型在真实数据匮乏的科学与工业领域实现性能提升。
原文摘要 · Abstract (English)
Predictive modeling in engineering applications has long been dominated by bespoke models and small, siloed tabular datasets, limiting the applicability of large-scale learning approaches. Despite recent progress in tabular foundation models, the resulting synthetic training distributions used for pre-training may not reflect the statistical structure of engineering data, limiting transfer to engineering regression. We introduce TREDBench, a curated collection of 83 real-world tabular regression datasets with expert engineering/non-engineering labels, and use TabPFN 2.5's dataset-level embedding to study domain structure in a common representation space. We find that engineering datasets are partially distinguishable from non-engineering datasets, while standard procedurally generated datasets are highly distinguishable from engineering datasets, revealing a substantial synthetic-real domain gap. To bridge this gap without training on real engineering samples, we propose an embedding-guided synthetic data curation method: we generate and identify "engineering-like" synthetic datasets, and perform continued pre-training of TabPFN 2.5 using only the selected synthetic tasks. Across 35 engineering regression datasets, this synthetic-only adaptation improves predictive accuracy and data efficiency, outperforming TabPFN 2.5 on 29/35 datasets and AutoGluon on 27/35, with mean multiplicative data-efficiency gains of 1.75x and 4.44x, respectively. More broadly, our results indicate that principled synthetic data curation can convert procedural generators into domain-relevant "data engines," enabling foundation models to improve in data-sparse scientific and industrial domains where real data collection is the primary bottleneck.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。