为表格模型设计轻量文本适配器，避免嵌入压缩损失。
Towards Pretraining Text Encoders for TabPFN
- 用轻量适配器将文本嵌入映射到表格模型的嵌入空间。
- 相比传统方法，保留更多文本信息且训练效率更高。
- 适合需要融合文本与表格数据的高效建模场景。
表格基础模型（如 TabPFN）在包含数值和类别数据的表格数据集上表现优异，但无法原生处理高基数文本特征。现有流程通常使用语言模型对文本进行嵌入，并通过 PCA 将向量压缩为少量标量特征后输入 TabPFN，造成信息瓶颈：大部分嵌入维度被丢弃，且压缩表示需由 TabPFN 的特征编码器重新展开。端到端替代方案可避免 PCA，但需大量含文本单元的预训练数据，且性能通常不如在大规模合成数据上预训练的表格基础模型。受多模态对齐方法（如 LLaVA 和 TableGPT）启发，我们提出 TabPFN 文本适配器（text-to-TFM token projection）。冻结句子编码器和 TabPFN，仅训练一个轻量级适配器，将文本嵌入映射为 TabPFN 嵌入空间中的短序列标记。该设计消除 PCA 瓶颈，保留 TabPFN 的数值优势，且训练效率高于端到端文本-表格管道。
原文摘要 · Abstract (English)
Tabular foundation models, such as TabPFN, achieve strong performance on tabular datasets with numerical and categorical data, but do not natively handle high-cardinality text features. Standard pipelines, therefore, embed text with a language model and compress the resulting vectors with PCA into a small number of scalar features before inputting them into TabPFN. This creates an information bottleneck: most embedding dimensions are discarded, and the compressed representation must then be expanded again by TabPFN's feature encoder. End-to-end alternatives can avoid PCA, but they require large amounts of pretraining data containing text cells and usually perform subpar compared to tabular foundation models that were pretrained on large amounts of synthetic data. Inspired by modality-alignment approaches like LLaVA (vision-to-LLM token projection) and TableGPT-style systems (table-to-LLM token projection), we introduce the TabPFN Text Adapter (text-to-TFM token projection). We freeze both the sentence encoder and TabPFN, and train only a lightweight adapter that maps text embeddings into a short sequence of tokens in TabPFN's embedding space. This design removes the PCA bottleneck, preserves TabPFN's numerical strengths, and is more efficient to train than end-to-end text-tabular pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。