TabICLv2是更快更省显存的表格基础模型,无需调参即超越现有最佳表现。
TabICLv2: A better, faster, scalable, and open tabular foundation model
- 用新生成引擎提升预训练数据多样性,配合架构优化增强泛化能力。
- 在百万级数据上仅用50GB显存即可有效推理,速度显著快于当前最优模型。
- 开源全部推理代码与权重,适合追求高效部署的工业研究者使用。
表格基础模型如TabPFNv2和TabICL近期在预测基准上超越梯度提升树,证明了上下文学习对表格数据的价值。我们提出TabICLv2,一种新的回归与分类顶尖基础模型,基于三大支柱:(1) 针对高预训练多样性的新型合成数据生成引擎;(2) 多项架构创新,包括一种新的可扩展注意力软最大函数,提升对大规模数据集的泛化能力,且无需代价高昂的长序列预训练;(3) 优化的预训练协议,尤其以Muon优化器替代AdamW。在TabArena与TALENT基准上,未进行任何调参的TabICLv2已超越当前最优模型RealTabPFN-2.5(需超参数调优、集成及真实数据微调)。仅需适度预训练计算资源,TabICLv2可在50GB GPU内存下有效处理百万级数据集,且明显快于RealTabPFN-2.5。我们通过大量消融实验量化各项贡献,并率先开放推理代码与模型权重至https://github.com/soda-inria/tabicl,合成数据生成引擎与预训练代码随后发布。
原文摘要 · Abstract (English)
Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and commit to open research by first releasing inference code and model weights at https://github.com/soda-inria/tabicl, with synthetic data engine and pretraining code to follow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。