混合注意力与状态空间模型，提升表格数据推理效率与性能。
Tydra: An Efficient Hybrid Model for Tabular Data
- 交替使用注意力与状态空间层，兼顾计算效率与建模能力。
- 在30个OpenML数据集上，推理速度比TabPFN快30%。
- 性能接近大10倍的Hydra模型，适合高效表征学习场景。
基于Transformer的表格基础模型（如TabPFN）虽具备强大预测能力，但计算开销随上下文长度呈二次增长。而亚二次复杂度的状态空间模型（SSM）方法（如Hydra）则以牺牲精度换取效率。为平衡二者，本文提出Tydra——一种融合Transformer与状态空间模型的混合架构，通过交错排列注意力与SSM层实现上下文学习。在30个OpenML数据集上，Tydra相较TabPFN将推理时间缩短30%，同时保持接近其性能；且优于一个约大十倍的Hydra模型，兼具更快推理速度与更高准确率。结果表明，混合架构是表格基础模型的重要发展方向。
原文摘要 · Abstract (English)
Transformer-based tabular foundation models such as TabPFN achieve strong predictive performance but incur quadratic computational cost with context length. On the other hand, subquadratic SSM-based alternatives such as Hydra trade away accuracy for efficiency. To balance both, we introduce Tydra, a hybrid Transformer-State Space Model (SSM) architecture for tabular in-context learning that interleaves attention and SSM layers. Across 30 OpenML datasets, Tydra reduces inference time by 30% relative to TabPFN while retaining much of its predictive performance. Tydra also outperforms an approximately ten-times-larger Hydra model while providing faster inference. The results indicate that hybrid architectures are a promising direction for tabular foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。