提出OpenRFM,解决关系型模型少样本学习的性能瓶颈
OpenRFM: Dissecting Relational In-Context Learning
- 设计双阶段架构,融合关系主干与批量少样本层
- 合成数据+持续真实数据预训练,提升30%平均性能
- 适合需要高效处理关系数据的工业级少样本学习场景
关系基础模型(RFMs)承诺仅用一个预训练预测器,通过关系型上下文学习(ICL)在给定任意关系数据库时实现单次前向传播预测。然而,开源RFMs与商业版本之间存在显著差距,其根源尚未系统理解。本文从模型和数据两方面剖析代表性框架——关系变压器(RT)。模型层面:发现RT执行的是关系级ICL,核回归视角显示当标签单元覆盖稀疏导致回归欠定则失败。数据层面:消融实验表明,仅用合成数据或分布内预训练会使相同架构进入不同状态——懒惰态与特征学习态。进一步分析揭示缺失关键在于标签生成过程中支持可识别的关系潜在表示。据此提出(1)双阶段ICL架构,结合关系主干与来自预训练表格模型的批量级ICL层以缓解关系级标签稀缺;(2)融合同质性感知的合成数据与持续真实数据预训练,并引入基于原型的正则化。这些设计构成OpenRFM,一个简单但有效的RFM,在大量评估任务上相比RT主干平均性能提升约30%,并超越商业模型KumoRFMv1。
原文摘要 · Abstract (English)
Relational Foundation Models (RFMs) promise a single pre-trained predictor that, given any relational database, returns predictions in one forward pass via relational in-context learning (ICL). Yet a substantial gap separates open RFMs from their commercial counterparts, and the origin of this gap has not been systematically understood. We dissect a representative framework, the Relational Transformer (RT), from two perspectives. Model side: we show that RT performs relation-level ICL, and a kernel regression view shows it fails when sparse label-cell coverage yields an underdetermined regression. Data side: we ablate RT's pre-training source and find that existing synthetic-only pre-training and in-distribution pre-training drive the same architecture into different regimes, lazy vs. feature-learning. Probing this gap reveals that the missing ingredient is a support-identifiable relational latent in the label-generation process. These two diagnoses translate into (1) a dual-stage ICL architecture that combines the relational backbone with a batch-level ICL layer lifted from a pre-trained tabular foundation model to overcome relation-level label scarcity, and (2) a homophily-aware synthetic plus continual real-data pre-training mixture, augmented with a prototype-based regularization. These choices define OpenRFM, a simple yet effective RFM that improves average task performance by approximately 30% over the RT backbone and surpasses the commercial model KumoRFMv1 on a large set of evaluation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。