无需微调,用合成数据训练的表格大模型实现高精度预测。
Xiaomi-TabLDM: A Tabular Foundation Model Technical Report

- 基于因果生成的合成数据预训练,支持上下文学习
- 多个基准上排名第一或第二,训练与推理耗时更低
- 测试时可扩展计算量,提升预测性能,适合高效部署
我们提出 Xiaomi-TabLDM,一个通过上下文学习进行分类与回归的表格基础模型,无需任务微调即可实现优异预测精度。模型仅在结构因果模型(SCMs)生成的合成数据上预训练,具备更灵活的上下文利用能力与更高效的容量扩展性。一)新性能标准:在 OpenML-CTR23 上排名首位,在 TALENT、TabArena 与 BCCO 的回归任务中均位列第二,跨四个互补基准表现一致出色;同时具备良好的性能-效率权衡:在 TabArena 回归任务中,以第二高 Elo 分数,实现比榜首模型少 82% 训练时间与 68% 预测时间的显著效率优势。二)大规模合成预训练:拓展了用于预训练的合成表格数据覆盖范围与多样性;采用三阶段训练策略,结合双流特征分组、轻量级注意力残差与稀疏专家混合(sparse Mixture-of-Experts),使模型能学习更丰富的特征交互与任务专业化。三)测试时扩展:通过在推理时增加计算量,持续提升基线模型的预测性能,进一步扩展表格预测能力。
原文摘要 · Abstract (English)
We introduce Xiaomi-TabLDM, a tabular large data foundation model for classification and regression via in-context learning, which delivers superior prediction accuracy without requiring task-specific fine-tuning. Pretrained exclusively on synthetic data generated from structural causal models (SCMs), our model enables more flexible context utilization and more efficient capacity scaling. i) A new performance standard. Strong regression performance across benchmarks: Xiaomi-TabLDM ranks 1st on OpenML-CTR23 and 2nd on regression across TALENT, TabArena, and BCCO, demonstrating consistently strong regression performance across four complementary benchmark suites. Favorable performance--efficiency trade-off: Xiaomi-TabLDM combines strong predictive performance with substantially lower computational cost. For example, on TabArena regression, it achieves the second-highest Elo while using 82% less training time and 68% less prediction time than the top-ranked TabFM. ii) Large-scale synthetic pretraining. Xiaomi-TabLDM expands the coverage and diversity of synthetic tabular data used for pretraining. We also adopt a three-stage training strategy together with dual-stream feature grouping, lightweight Attention Residual, and sparse Mixture-of-Experts, enabling Xiaomi-TabLDM to learn richer feature interactions and expert specialization across diverse tabular tasks. iii) Test-time scaling. Xiaomi-TabLDM further extends tabular prediction through test-time compute scaling, where allocating additional computation at inference time consistently improves predictive performance over the base model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。