通过改进令牌化与路由机制,提升表格模型的效率与精度
LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models

- 用RBF局部特征扩展标量,增强浅层表达能力
- 200万参数模型在多个基准上超越更大模型
- 适合追求高效高精度表格建模的研究者
表格基础模型(TFMs)性能日益接近树集成方法,但计算效率低下:标准仿射标量令牌化使每个特征仅通过一维通道注入值变化,特征ID与位置信号无法提升特征内值自由度,导致早期层敏感性弱、隐藏状态冗余。本文提出统一的令牌化与路由框架:RaBEL将标量扩展为紧凑的局部RBF特征(可选指数门控),改善条件性和浅层有效秩;重组双向块S→N→F,在特征混合前聚合跨样本上下文并使用注意力池化,对齐读出过程。二者结合形成LimiX-2M,一个200万参数模型,在广泛使用的表格基准上优于更大的TabPFN-v2和TabICL基线,同时降低训练与推理成本。结果表明,价值感知令牌化与读出对齐路由是提升TFMs准确率-效率权衡的关键。模型检查点与推理代码见https://github.com/limix-ldm-ai/LimiX。
原文摘要 · Abstract (English)
Tabular foundation models (TFMs) increasingly rival tree ensembles, but their performance is often compute-inefficient: with standard affine scalar tokenization, each feature injects value variation through an essentially one-dimensional channel, and feature IDs/positional signals cannot increase within-feature value degrees of freedom, yielding weak early-layer value sensitivity and redundant hidden states. We present a unified tokenize-and-route framework for strong TFMs: RaBEL expands each scalar into compact localized RBF features (optionally exponent-gated) to improve conditioning and shallow-layer effective rank, while a reordered bidirectional block S->N->F aligns computation with the readout by aggregating cross-sample context before feature mixing and using attention pooling. Together, these changes yield LimiX-2M, a 2M-parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used tabular benchmarks while reducing training and inference costs. These results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs. Model checkpoints and inference code are available at https://github.com/limix-ldm-ai/LimiX.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。