提出可保证稳定性的Transformer架构，让模型更可靠。
Approximation Theory for Lipschitz Continuous Transformers
- 用梯度下降思想设计注意力和MLP模块，天然满足稳定性。
- 证明该类模型能在限制光滑性的空间中逼近任意函数。
- 分析不依赖词元数量，适合安全关键场景使用。
在安全敏感场景中，稳定性和鲁棒性至关重要。一种有原则的方法是约束模型的Lipschitz常数。然而，对显式保持Lipschitz连续性的架构，尚无近似理论保证。本文引入一类基于梯度下降的上下文Transformer，其设计保证了Lipschitz连续性。我们通过将MLP和注意力模块实现为负梯度流的显式欧拉步，确保内在稳定性而不牺牲表达能力。我们在一个受Lipschitz约束的函数空间内证明了通用近似定理。关键的是，我们的分析采用测度论形式，将Transformer视为概率测度上的算子，从而获得与词元数量无关的近似保证。这些结果为设计稳健、Lipschitz连续的Transformer架构提供了严格的理论基础。
原文摘要 · Abstract (English)
Stability and robustness are critical for deploying Transformers in safety-sensitive settings. A principled way to enforce such behavior is to constrain the model's Lipschitz constant. However, approximation-theoretic guarantees for architectures that explicitly preserve Lipschitz continuity have yet to be established. In this work, we bridge this gap by introducing a class of gradient-descent-type in-context Transformers that are Lipschitz-continuous by construction. We realize both MLP and attention blocks as explicit Euler steps of negative gradient flows, ensuring inherent stability without sacrificing expressivity. We prove a universal approximation theorem for this class within a Lipschitz-constrained function space. Crucially, our analysis adopts a measure-theoretic formalism, interpreting Transformers as operators on probability measures, to yield approximation guarantees independent of token count. These results provide a rigorous theoretical foundation for the design of robust, Lipschitz continuous Transformer architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。