arXiv:2512.09530stat.MLcs.LG2025-12

用最优传输理论分析自注意力训练,提出更高效的表格数据分类新方法

Transformers for Tabular Data: A Training Perspective of Self-Attention via Optimal Transport

  • 通过最优传输度量追踪自注意力投影演化过程
  • 新方法在标准输入下准确率媲美Transformer且计算成本更低
  • 适合关注模型训练机制与表格数据高效建模的研究者

本论文从最优传输(OT)视角审视自注意力的训练过程,提出一种面向表格分类的OT基替代方案。研究通过离散OT指标(如Wasserstein距离、Monge间隙、最优性与效率)追踪训练中自注意力层的中间投影演化。实验涵盖二类与三类分类任务,以及一个生物医学数据集。结果表明,最终自注意力映射常逼近OT最优耦合,但训练轨迹效率低下。对MLP模块进行合成数据预训练可部分提升收敛速度,但对初始化敏感。为此,本文提出新算法:生成类别相关的虚拟高斯分布,计算其与真实数据的OT对齐,并训练MLP以泛化该映射。该方法在标准化输入下达到与Transformer相当的准确率,同时降低计算开销并具备更好扩展性,但性能依赖于虚拟数据几何设计的合理性。所有实验与实现均基于R语言完成。

原文摘要 · Abstract (English)

This thesis examines self-attention training through the lens of Optimal Transport (OT) and develops an OT-based alternative for tabular classification. The study tracks intermediate projections of the self-attention layer during training and evaluates their evolution using discrete OT metrics, including Wasserstein distance, Monge gap, optimality, and efficiency. Experiments are conducted on classification tasks with two and three classes, as well as on a biomedical dataset. Results indicate that the final self-attention mapping often approximates the OT optimal coupling, yet the training trajectory remains inefficient. Pretraining the MLP section on synthetic data partially improves convergence but is sensitive to their initialization. To address these limitations, an OT-based algorithm is introduced: it generates class-specific dummy Gaussian distributions, computes an OT alignment with the data, and trains an MLP to generalize this mapping. The method achieves accuracy comparable to Transformers while reducing computational cost and scaling more efficiently under standardized inputs, though its performance depends on careful dummy-geometry design. All experiments and implementations are conducted in R.

自注意力最优传输表格数据模型效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。