用线性注意力让表格模型秒级处理百万数据，效率超传统方法。
TabFlex: Scaling Tabular Learning to Millions with Linear Attention
- 引入线性注意力替代复杂度高的自注意力，提升可扩展性。
- 百万样本扑克牌数据集5秒内完成推理，速度比TabPFN快2倍以上。
- 适合需要快速部署、处理海量表格数据的工业场景。
利用大语言模型的上下文学习能力进行表格分类受到广泛关注，因其无需训练即可适配多种数据集。近期进展如TabPFN在小规模表格数据上表现优异，但在大规模复杂数据上难以扩展。本文通过引入线性注意力机制，作为复杂度为二次方的自注意力的高效替代方案，显著提升了TabPFN的效率与可扩展性。所提出的模型TabFlex能高效处理含数千特征、数百类别及数百万样本的表格数据。例如，在包含超过一百万样本的poker-hand数据集上，仅需5秒即可完成推理。大量实验表明，TabFlex相比TabPFN实现2倍以上加速，较XGBoost提升1.5倍，优于25种基准方法。此外，结合降维与采样等数据高效技术后，其在大规模数据上仍保持高性能且计算成本大幅降低。
原文摘要 · Abstract (English)
Leveraging the in-context learning (ICL) capability of Large Language Models (LLMs) for tabular classification has gained significant attention for its training-free adaptability across diverse datasets. Recent advancements, like TabPFN, excel in small-scale tabular datasets but struggle to scale for large and complex datasets. Our work enhances the efficiency and scalability of TabPFN for larger datasets by incorporating linear attention mechanisms as a scalable alternative to complexity-quadratic self-attention. Our model, TabFlex, efficiently handles tabular datasets with thousands of features and hundreds of classes, scaling seamlessly to millions of samples. For instance, TabFlex processes the poker-hand dataset with over a million samples in just 5 seconds. Our extensive evaluations demonstrate that TabFlex can achieve over a 2x speedup compared to TabPFN and a 1.5x speedup over XGBoost, outperforming 25 tested baselines in terms of efficiency across a diverse range of datasets. Furthermore, TabFlex remains highly effective on large-scale datasets, delivering strong performance with significantly reduced computational costs, especially when combined with data-efficient techniques such as dimensionality reduction and data sampling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。