用分阶段合成数据训练,小样本也能让模型学会复杂关系推理。
Curriculum Matters: Data-Efficient Relational PFN Pretraining with Synthetic Data
- 分阶段逐步增加表结构复杂度,提升模型关系推理能力
- 仅用1.3万张合成表就达到0.703的平均性能,比之前少45倍数据
- 单表预训练模型直接用于关系任务,性能接近专用管道
关系先验-数据拟合网络(PFNs)如RDB-PFN通过在数百万个合成任务上预训练来近似多表关系数据库上的贝叶斯推断。本文研究三个核心问题:第一,能否用结构不同的合成生成器PluRel替代RDB-PFN的先验?第二,合成数据的呈现顺序如何影响下游性能?第三,单表合成预训练能带来多少关系推理能力?以PluRel作为唯一合成数据源,发现:(i) 从7到17列逐步扩展的单表课程训练,在仅约13,300张合成表(比RDB-PFN少45倍单表数据集)下,23任务表格基准平均ROC-AUC达0.703;而一次性全部训练则降至0.541;(ii) 从零开始仅用约5,500个PluRel数据库的关系课程训练,19任务RelBench/4DBInfer基准平均ROC-AUC达0.638,恢复了RDB-PFN报告性能的88%,且仅需约220倍更少的关系合成数据;(iii) 单表课程模型未经任何关系适应,直接在关系基准上评估即达0.631,几乎匹配专用关系流水线。结果表明,课程设计与合成数据多样性对关系型PFN预训练的重要性,可能超过特定生成器或合成数据规模本身。
原文摘要 · Abstract (English)
Relational Prior-Data Fitted Networks (PFNs) such as RDB-PFN approximate Bayesian inference over multi-table relational databases by pretraining on millions of synthetic tasks. We investigate three intertwined questions about this paradigm. First, can a structurally different synthetic generator PluRel substitute for RDB-PFN's prior? Second, how much does the order in which synthetic data is presented to the PFN affect downstream performance? Third, how much relational reasoning can a PFN acquire from single-table synthetic pretraining alone, before any relational data is introduced? Using PluRel as the sole synthetic data source across all experiments, we find: (i) a progressive single-table curriculum that gradually widens schema complexity from 7 to 17 columns reaches 0.703 average ROC-AUC on the 23-task tabular benchmark using only approximately 13,300 synthetic tables (approximately 45x fewer single-table datasets than RDB-PFN's reported warm-up recipe), while the same data trained all-at-once collapses to 0.541 ROC-AUC; (ii) a relational curriculum trained from scratch on only approximately 5,500 PluRel databases reaches 0.638 average ROC-AUC on the 19-task RelBench/4DBInfer benchmark, recovering 88% of RDB-PFN's reported performance with approximately 220x less relational synthetic data; and (iii) the single-table curriculum model, evaluated directly on the relational benchmark without any relational adaptation, achieves 0.631, nearly matching the dedicated relational pipeline. Together, these findings suggest that curriculum design and synthetic data diversity may matter more for relational PFN pretraining than the specific relational generator or raw synthetic scale alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。