用外部生成数据训练关系型模型,少用90%数据仍达原性能的87%。
PluRel-to-RDB-PFN: Schema-Guided Synthetic Relational Pretraining
- 用PluRel生成数据库,通过课程策略融入RDB-PFN训练流程
- 仅5500个库3.3万任务,1024轮上下文时达0.6346的平均AUC
- 早期接触真实模式比后期适应更有效,适合高效预训练研究者
关系基础模型(RFMs)需要大规模合成关系型数据库进行预训练,但现有方法将数据生成与模型训练紧密耦合。本文研究通用合成关系数据库生成器PluRel是否可作为RDB-PFN的外部数据源。RDB-PFN原采用60万任务单表预热和约180万任务适配阶段。我们构建转换管道,将PluRel生成的数据库及外部二分类任务映射为RDB-PFN训练格式,并评估三种课程策略:先真实模式后合成(SCHEMA-GUIDED FIRST)、全程合成(FULLY SYNTHETIC)、先合成后真实模式(SCHEMA-GUIDED LAST)。仅使用约5500个关系型数据库(约3.3万任务),远低于原协议的55倍,且无单表预热,最佳策略(SCHEMA-GUIDED FIRST)在1024样本上下文中实现19个真实基准任务平均ROC-AUC为0.6346,恢复原性能(0.7245)的87.6%;在64样本上下文中差距缩小至93.8%(0.6116 vs. 0.6517)。结果表明,结合合理课程设计,外部合成生成器可提供有效预训练信号,且早期接触真实模式显著优于后期适应。
原文摘要 · Abstract (English)
Relational Foundation Models (RFMs) require large-scale synthetic relational databases for pretraining, but existing approaches tightly couple data generation with the model training pipeline. We study whether PluRel, a general-purpose synthetic relational database generator, can serve as an external data source for RDB-PFN, a relational in-context learner originally pretrained with a 600K-task single-table warm-up followed by an approximately 1.8M-task adaptation stage. We build a conversion pipeline that maps PluRel-generated databases, including externally constructed binary prediction tasks, into the RDB-PFN training format and evaluate three curriculum strategies: SCHEMA-GUIDED FIRST (real-world schema then fully synthetic), FULLY SYNTHETIC (diverse synthetic schemas throughout), and SCHEMA-GUIDED LAST (fully synthetic then real-world schema). Using only approximately 5,500 relational databases (approximately 33K tasks), roughly 55x fewer tasks than the original protocol, and no single-table warm-up, our best curriculum (SCHEMA-GUIDED FIRST) achieves 0.6346 average ROC-AUC across 19 real benchmark tasks at 1024-shot context, recovering 87.6% of the published RDB-PFN performance (0.7245). At 64-shot context, the gap narrows to 93.8% (0.6116 vs. 0.6517). Our results demonstrate that external synthetic generators can provide useful pretraining signals for RFMs when combined with appropriate curriculum design and that exposure to a real-world schema early in training is substantially more effective than late-stage schema adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。