对比真实与合成表格数据分布,发现合成数据存在显著偏差。
Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models

- 用特征统计和判别器评估三类表格数据分布差异
- 合成数据仅覆盖真实数据的狭小区间,优化参数无法改善
- 真实数据间分布相似,适合构建通用表格模型
表格基础模型在三种类型的语料库上预训练:来自基准库的精选数据集、从网络大规模抓取的表格,以及从参数化生成先验采样的合成表格。尽管预训练数据对模型性能至关重要,但这些语料库之间的分布关系尚不明确。本文选取三类典型数据集:代表网络抓取的T4、代表精选表格的TabFM,以及唯一公开参数的合成先验TabICL。通过表级、列级及相关性等聚合特征进行分布刻画,并使用判别器AUC和k-NN覆盖度量进行比较。结果表明,TabICL合成先验仅占据真实表格空间的狭窄区域,即使在超过86,000种超参数配置下优化也无法弥合这一差距;而精选与网络抓取数据在特征空间中分布相近,可互换使用。令人意外的是,合成数据与真实数据间的分布差距,在基于特征的接近度或TabICL自身内部表示中均未表现出明显性能影响,暗示分布覆盖并非其泛化能力的主要驱动因素。
原文摘要 · Abstract (English)
Tabular foundation models are pre-trained on one of three classes of corpus: curated datasets drawn from benchmark repositories, tables harvested at scale from the web, or synthetic tables sampled from a parametric generative prior. Despite the centrality of pre-training data to model performance, little is known about how these corpora relate to one another in distribution, and the impact this has on downstream performance. In this work we take three canonical, archetypal datasets used to train tabular foundation models; the T4 dataset represents web-scraped corpora, the TabFM dataset curated tables from Kaggle, and the TabICL dataset as the only well-used synthetic prior with publicly available parameters. We characterise each corpus using aggregate features over whole tables, columns and correlations, and compare them using discriminator AUCs and k-NN coverage metrics. We find that the TabICL synthetic prior occupies a narrow region of the space of real tables, that this mismatch cannot be closed by optimising prior hyper-parameters across more than 86 thousand configurations, and that curated and web-scraped corpora are broadly interchangeable on a distributional level in feature space. Surprisingly, the distributional gap between synthetic pre-training data and real tables has a clearly detectable effect on performance under neither feature-based proximity measures or TabICL's own internal representations, suggesting that coverage of the real-data distribution is not the primary driver of TabICL's generalisation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。