揭示了Transformer通过深度与提示工程实现最优传输对齐的机制。
Provable optimal transport with transformers: The essence of depth and prompt engineering
- 用梯度下降模拟熵正则化最优传输的对偶问题，解释注意力机制
- 证明模型深度决定对齐精度，深度增加可逼近最优传输
- 适用于理解大模型对齐机制，适合研究者和算法工程师
尽管在实践中表现优异，但变压器模型在语言处理中如何对齐词元的内在机制仍不清楚。本文提供了对大型语言模型中词元对齐的机制与理论解释。我们首先通过实证证据表明，在机器翻译任务中，注意力权重随层数递进地对齐翻译词对，其过程接近词嵌入间的最优传输（OT）。基于此观察，我们证明了softmax自注意力层可模拟熵正则化最优传输对偶问题的梯度下降，为对齐提供了理论基础。分析进一步给出一个构造性收敛界，表明模型深度控制最优传输近似的精度。直接推论是：标准变压器无需参数调整即可对任意长度列表排序，误差项随模型深度趋于零。
原文摘要 · Abstract (English)
Despite their empirical success, the internal mechanism by which transformer models align tokens during language processing remains poorly understood. This paper provides a mechanistic and theoretical explanation of token alignment in LLMs. We first present empirical evidences showing that, in machine translation, attention weights progressively align translated word pairs across layers, closely approximating Optimal Transport (OT) between word embeddings. Building on this observation, we prove that softmax self-attention layers can simulate gradient descent on the dual of the entropy-regularized OT problem, providing a theoretical foundation for the alignment. Our analysis yields a constructive convergence bound showing that transformer depth controls OT approximation accuracy. A direct implication is that standard transformers can sort lists of varying lengths without any parameter adjustment, up to an error term vanishing with transformers depth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。