用大模型处理表格数据，解决数字解析和上下文不足问题
TabGemma: Text-Based Tabular ICL via LLM using Continued Pretraining and Retrieval
- 将数值转为科学记数法，用真实数据继续预训练120亿参数模型
- 通过短语检索在128k令牌内选出关键样本，分类任务表现超越现有方法
- 适合需要高语义理解的表格预测场景，尤其小样本情况优势明显
我们研究基于混合文本、数值和类别字段的表格预测任务。提出TabGemma，一种无需依赖模式结构的上下文学习方法，将行视为序列，并解决预训练大模型用于表格预测时的两个实际挑战：不稳定的数值分词与有限上下文长度。通过使用带符号的科学记数法统一数值表示，并利用大规模真实世界数据对120亿参数的Gemma 3模型进行目标填充式的持续预训练。推理阶段采用紧凑的n-gram检索机制，筛选出能在128k令牌窗口内容纳的有信息量的示例。在语义丰富的基准测试中，TabGemma在低数据和高数据场景下的分类任务均达到新最佳性能，并随上下文行数增加而持续提升；回归任务在小样本下表现竞争力，但随着数据增长落后于传统方法。结果表明，当结合专用数值处理与上下文检索时，大模型可在高度语义化的任务中成为有效的表格上下文学习器，同时推动了数值建模与长上下文扩展的进一步发展。
原文摘要 · Abstract (English)
We study LLMs for tabular prediction with mixed text, numeric, and categorical fields. We introduce TabGemma, a schema-agnostic in-context learner that treats rows as sequences and tackles two practical hurdles when adapting pretrained LLMs for tabular predictions: unstable numeric tokenization and limited context size. We propose to canonicalize numbers via signed scientific notation and continue pretraining of a 12B Gemma 3 model with a target imputation objective using a large-scale real world dataset. For inference, we use a compact n-gram-based retrieval to select informative exemplars that fit within a 128k-token window. On semantically rich benchmarks, TabGemma establishes a new state of the art on classification across low- and high-data regimes and improves monotonically with more context rows. For regression, it is competitive at small sample sizes but trails conventional approaches as data grows. Our results show that LLMs can be effective tabular in-context learners on highly semantic tasks when paired with dedicated numeric handling and context retrieval, while motivating further advances in numeric modeling and long-context scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。