高效表格预测模型，速度远超同类，性能不降。
TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction
- 采用行注意力+长上下文预训练，无需检索提升效率。
- 在三个数据集上性能接近v1.1，推理速度提升数个数量级。
- 适合对速度敏感的实时表格预测场景。
基于上下文学习的表格基础模型发展迅速，但现有方法或依赖单元格架构或检索机制，牺牲效率换取性能，限制了在计算资源有限或需快速推理场景的应用。本文提出一种新方法：保持行级注意力结构，通过长上下文预训练消除检索需求。结合架构优化及在新收集更大规模真实数据上的自监督预训练，构建出TabDPT-Turbo模型。该模型在TabArena-Lite、CC18和CTR23三个基准上表现与TabDPT v1.1相当，但推理速度显著更快，在主流基础模型中整体最快。相关代码已发布于https://github.com/layer6ai-labs/TabDPT-inference，版本为TabDPT v1.2。
原文摘要 · Abstract (English)
Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity. However, recent approaches with either cell-based architectures or retrieval have sacrificed efficiency for raw performance, restricting their utility in situations where compute is limited or inference speed is crucial. We adopt an alternate approach, sticking with row-based attention while incorporating long context pre-training to eliminate the need for retrieval. By combining this with architectural improvements and SSL pre-training on a newly-sourced, larger corpus of real data results, we present TabDPT-Turbo, a model that provides comparable default performance to TabDPT v1.1 on TabArena-Lite, CC18, and CTR23, at orders of magnitude faster. In our experiments, TabDPT-Turbo is the fastest model overall among leading foundation models. We have released the new model as TabDPT v1.2 at https://github.com/layer6ai-labs/TabDPT-inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。