一个模型搞定表格分类与回归,性能超越主流工具。
TabH2O: A Unified Foundation Model for Tabular Prediction

- 统一架构+单阶段预训练,同时处理分类与回归任务。
- 在300个数据集上平均排名2.54,80%任务进入前三。
- 抗噪声设计,适合百万级行、百特征的表格数据。
我们提出TabH2O,一个面向表格数据的统一基础模型,通过上下文学习在单次前向传播中完成分类与回归任务。该模型基于TabICL架构,进行三项关键改进:(1)统一训练,采用双头结构使单一模型同时处理两类任务,无需独立模型,降低预训练成本;(2)单阶段预训练,引入有界可扩展softmax、阶段间归一化、可学习残差缩放和对数输出软截断等技术,提升训练稳定性,无需分阶段课程学习,可从完整序列开始训练;(3)噪声感知预训练,合成数据集显式引入噪声维度,增强模型对无关特征的鲁棒性。我们在TALENT基准(300个数据集)上评估TabH2O v1.1(2920万参数),平均排名2.54,优于调优后的CatBoost(4.00)、H2O AutoML(4.28)、LightGBM(4.98)和TabPFN v2.6(2.80),接近TabICL v2(2.13),在分类与回归任务中80%的测试集表现位列前三。在TabArena上也达到领先水平。该模型适用于约50万行、100维特征以内的表格数据。
原文摘要 · Abstract (English)
We present TabH2O, a foundation model for tabular data that performs classification and regression in a single forward pass via in-context learning. TabH2O builds on the TabICL architecture with several key modifications: (1) unified training, a single model handles both classification and regression via a dual-head architecture, eliminating the need for separate models and reducing total pretraining cost; (2) single-stage pretraining, training stability improvements (bounded scalable softmax, inter-stage normalization, learnable residual scaling, logit soft-capping) eliminate the need for multi-stage curriculum learning, enabling training with full-length sequences from the start; and (3) noise-aware pretraining, synthetic datasets include explicit noise dimensions to teach the model robustness to irrelevant features. We evaluate TabH2O v1.1 (29.2M parameters) on the TALENT benchmark (300 datasets), where it achieves an average rank of 2.54 out of 6 evaluated methods, outperforming tuned CatBoost (4.00), H2O AutoML (4.28), LightGBM (4.98), and TabPFN v2.6 (2.80), and competitive compared to TabICL v2 (2.13), while placing in the top-3 on 80% of the testing datasets across classification and regression tasks. We also achieve state-of-the-art results on TabArena. Moreover, our single model is well-suited to tabular datasets up to roughly 500,000 rows and 100 features.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。