用线性时序状态空间模型替代Transformer，提升表格数据建模效率
State-Space Models for Tabular Prior-Data Fitted Networks
- 采用双向线性时序状态空间模型Hydra替代Transformer
- 减少输入顺序依赖，性能接近原始TabPFN模型
- 适合追求高效推理的表格数据场景
近年来，基于预训练Transformer架构的表格数据基础模型（如TabPFN）展示了其在高预测性能下近似贝叶斯推断的能力。然而，Transformer存在与序列长度呈二次复杂度的问题，促使人们探索更高效的序列模型。本文研究了Hydra——一种双向线性时序结构化状态空间模型（SSM）作为TabPFN中Transformer的替代方案。核心挑战在于SSM对输入标记顺序的高度敏感性，而表格数据的行顺序在语义上无意义。我们探讨双向方法在保持效率的同时，能否实现对称上下文聚合。实验表明，该方法有效降低了顺序依赖性，达到与原版TabPFN模型相当的预测性能。
原文摘要 · Abstract (English)
Recent advancements in foundation models for tabular data, such as TabPFN, demonstrated that pretrained Transformer architectures can approximate Bayesian inference with high predictive performance. However, Transformers suffer from quadratic complexity with respect to sequence length, motivating the exploration of more efficient sequence models. In this work, we investigate the potential of using Hydra, a bidirectional linear-time structured state space model (SSM), as an alternative to Transformers in TabPFN. A key challenge lies in SSM's inherent sensitivity to the order of input tokens - an undesirable property for tabular datasets where the row order is semantically meaningless. We investigate to what extent a bidirectional approach can preserve efficiency and enable symmetric context aggregation. Our experiments show that this approach reduces the order-dependence, achieving predictive performance competitive to the original TabPFN model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。