用最优传输理论改进Transformer，提升训练稳定性和泛化能力。
OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization
- 将Transformer建模为连续时间动力系统，通过最优传输正则化训练。
- 在文本、图像和点云任务中性能优于传统离散Transformer。
- 兼容现有架构，修改极少即可适配，适合需要稳定训练的场景。
Transformer在众多任务中取得领先性能。本文提出一种连续时间版Transformer，将变压器块作为动力系统的控制方程参数。利用最优传输理论对训练过程进行正则化，增强训练稳定性并提升模型泛化能力。理论上证明该正则化是必要的，可确保解的唯一性与光滑性。模型具有高度灵活性，几乎任意现有Transformer架构仅需微小代码修改即可构建该动力系统。我们在自然语言处理、图像分类和点云分类等任务上进行了大量数值实验，结果表明该方法在性能上优于其离散对应模型，并超越多个对比模型。
原文摘要 · Abstract (English)
Transformers have achieved state-of-the-art performance in numerous tasks. In this paper, we propose a continuous-time formulation of transformers. Specifically, we consider a dynamical system whose governing equation is parametrized by transformer blocks. We leverage optimal transport theory to regularize the training problem, which enhances stability in training and improves generalization of the resulting model. Moreover, we demonstrate in theory that this regularization is necessary as it promotes uniqueness and regularity of solutions. Our model is flexible in that almost any existing transformer architectures can be adopted to construct the dynamical system with only slight modifications to the existing code. We perform extensive numerical experiments on tasks motivated by natural language processing, image classification, and point cloud classification. Our experimental results show that the proposed method improves the performance of its discrete counterpart and outperforms relevant comparing models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。