用稀疏注意力提升表格数据学习效率，兼顾精度与速度。
TabNSA: Native Sparse Attention for Efficient Tabular Data Learning
- 引入原生稀疏注意力机制，动态聚焦关键特征子集。
- 在多个数据集上优于现有深度模型，少样本学习表现显著提升。
- 适合处理异构特征、小样本的表格数据任务。
表格数据因特征类型多样、缺乏空间结构且样本量有限，给深度学习带来挑战。本文提出TabNSA，将原生稀疏注意力（NSA）与TabMixer主干网络结合，高效建模表格数据。NSA模块采用分层稀疏注意力机制，包括令牌压缩、选择性保留和局部滑动窗口，显著降低标准注意力的二次复杂度，同时应对特征异质性。TabMixer通过并行独立参数的多层感知机分支捕捉复杂非线性依赖。两者通过逐元素相加与均值池化融合，实现全局上下文与细粒度交互的联合建模。在监督与迁移学习设置下广泛实验表明，TabNSA持续优于当前最优深度学习模型。进一步通过微调大语言模型（LLM）增强，实现语言引导的泛化，在多种表格基准上有效解决少样本学习问题。代码已开源。
原文摘要 · Abstract (English)
Tabular data poses unique challenges for deep learning due to its heterogeneous feature types, lack of spatial structure, and often limited sample sizes. We propose TabNSA, a novel deep learning framework that integrates Native Sparse Attention (NSA) with a TabMixer backbone to efficiently model tabular data. TabNSA tackles computational and representational challenges by dynamically focusing on relevant feature subsets per instance. The NSA module employs a hierarchical sparse attention mechanism, including token compression, selective preservation, and localized sliding windows, to significantly reduce the quadratic complexity of standard attention operations while addressing feature heterogeneity. Complementing this, the TabMixer backbone captures complex, non-linear dependencies through parallel multilayer perceptron (MLP) branches with independent parameters. These modules are synergistically combined via element-wise summation and mean pooling, enabling TabNSA to model both global context and fine-grained interactions. Extensive experiments across supervised and transfer learning settings show that TabNSA consistently outperforms state-of-the-art deep learning models. Furthermore, by augmenting TabNSA with a fine-tuned large language model (LLM), we enable it to effectively address Few-Shot Learning challenges through language-guided generalization on diverse tabular benchmarks. Code available on: https://github.com/aseslamian/TabNSA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。