ConTextTab融合语义理解与表格结构,提升真实数据上的表格上下文学习性能。
ConTextTab: A Semantics-Aware Tabular In-Context Learner
- 用多模态嵌入建模表格不同字段,增强语义感知能力
- 在真实世界表格数据上训练,于CARTE基准上刷新最佳成绩
- 兼顾高效架构与深度语义理解,适合需要高精度的工业级表格任务
表格上下文学习(ICL)近期在多个表格预测任务中达到领先水平。尽管当前表格式ICL架构在结构适配性和效率上表现优异,但其仅在合成数据上训练,难以充分利用真实表格中的丰富语义与世界知识。而基于预训练大模型如TabuLa-8B的模型虽具备深层语义理解能力,却受限于架构,只能使用少量上下文。为融合两者优势,我们提出ConTextTab,将语义理解与对齐机制融入表格式ICL框架。通过为不同数据模态设计专用嵌入,并在大规模真实世界表格数据上训练,该模型在广泛基准上表现媲美当前最优,且在语义丰富的CARTE基准上树立新标准。代码与模型检查点已公开。
原文摘要 · Abstract (English)
Tabular in-context learning (ICL) has recently achieved state-of-the-art (SOTA) performance on several tabular prediction tasks. Previously restricted to classification problems on small tables, recent advances such as TabPFN and TabICL have extended its use to larger datasets. Although current table-native ICL architectures are architecturally efficient and well-adapted to tabular data structures, their exclusive training on synthetic data limits their ability to fully leverage the rich semantics and world knowledge contained in real-world tabular data. At the other end of the spectrum, tabular ICL models based on pretrained large language models such as TabuLa-8B integrate deep semantic understanding and world knowledge but are only able to make use of a small amount of context due to inherent architectural limitations. With the aim to combine the best of both these worlds, we introduce ConTextTab, integrating semantic understanding and alignment into a table-native ICL framework. By employing specialized embeddings for different data modalities and by training on large-scale real-world tabular data, our model is competitive with SOTA across a broad set of benchmarks while setting a new standard on the semantically rich CARTE benchmark. Code and model checkpoints are available at: https://github.com/SAP-samples/sap-rpt-1-oss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。