arXiv:2603.16513cs.LGcs.AI2026-03

提出线性复杂度模型FEAT,高效处理超大规模结构化数据。

FEAT: A Linear-Complexity Foundation Model for Extremely Large Structured Data

  • 用双轴编码替代二次注意力,实现O(N)时间跨行上下文建模。
  • 在12个真实数据库上零样本任务表现优于现有模型,推理速度提升50倍。
  • 适合处理金融、医疗等含长尾分布的真实企业级结构化数据。

结构化数据广泛应用于医疗、金融和科学数据管理等领域。当前结构化数据基础模型(SFMs)在处理真实企业数据库时仍面临可扩展性和泛化能力挑战:多数SFMs依赖全自注意力机制,带来O(N²)计算瓶颈,限制了联合处理的数据量;直接用线性序列模型替代注意力可能破坏结构化数据的置换不变性,引入人工顺序偏差并降低表示质量。此外,仅在合成数据上训练的模型难以适应真实数据库中常见的长尾和异构分布。为此,我们提出FEAT,一种面向超大规模结构化数据的线性复杂度基础模型。FEAT采用多层双轴编码架构,以自适应融合双向状态空间模型(AFBM)与卷积门控线性注意力(Conv-GLA),在O(N)时间内实现跨行上下文建模,并支持置换不变表示学习。为增强对真实数据偏斜的鲁棒性,FEAT进一步采用混合结构因果预训练流程与稳健重建目标。在12个真实数据库基准上的实验表明,FEAT在零样本任务上持续优于代表性SFMs,且处理长度呈线性扩展,推理延迟最高降低50倍。

原文摘要 · Abstract (English)

Structured data is widely used in domains such as healthcare, finance, and scientific data management. Recent studies on structured data foundation models (SFMs) aim to support data analysis and mining tasks over such data, but still face scalability and generalization challenges when applied to real-world enterprise databases. First, many SFMs rely on full self-attention, which introduces an O(N^2) computational bottleneck and limits the number of tuples that can be processed jointly. Second, directly replacing attention with linear-complexity sequence models may conflict with the permutation-invariant nature of structured data, introducing artificial order bias and degrading representation quality. Moreover, models trained only on synthetic data may struggle to generalize to the heavy-tailed and heterogeneous distributions commonly found in real-world databases. To address these challenges, we propose FEAT, a linear-complexity foundation model for extremely large structured data. FEAT replaces quadratic attention with a multi-layer dual-axis encoding architecture. It integrates an adaptive-fusion bidirectional state-space model (AFBM) with convolutional gated linear attention (Conv-GLA), enabling cross-tuple contextualization in O(N) time while supporting permutation-invariant representation learning. To improve robustness under real-world data skewness, FEAT further adopts a hybrid structural causal pre-training pipeline with a robust reconstruction objective. Experiments on 12 real-world database benchmarks show that FEAT consistently outperforms representative SFMs on zero-shot tasks and scales linearly with structured-data sample length, achieving up to 50x faster inference latency.

结构化数据线性复杂度基础模型高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。