揭示了注意力机制本质是输入向量在自适应表面的投影。
Scaled Dot-Product Attention implements projection of inputs onto a common surface
- 将注意力重新表述为输入向量投影到自定义曲面，突破传统查询键值范式。
- 该投影能捕捉时变、上下文相关的非线性依赖关系。
- 适合研究时序数据中动态非线性模式的学者参考。
缩放点积注意力（SDPA）是大语言模型及其他非线性信号处理应用成功的关键组件。其理论基础长期依赖于源自数据库理论的“查询、键、值”概念，但这些概念难以与标准数学信号处理方法相协调。本文证明，SDPA可被重写为一种数学等价形式：将输入向量投影到由输入自身决定的共同曲面上。因此，SDPA能够发现依赖时间与上下文的非线性关系。这一新形式不仅提升了前向传播与学习算法的速度，更重要的是揭示了潜在扩展方向。在自然语言语境下，我们重新诠释了SDPA的作用：即通过输入向量所处的局部曲面，提取随时间变化的上下文语义。输入词嵌入据此被局部上下文曲面所修正。该解释与传统‘自注意力’概念有显著差异，并为处理具有时变局部非线性依赖的时间序列数据提供了强有力的理论支撑。
原文摘要 · Abstract (English)
Scaled dot-product attention (SDPA) is a fundamental component responsible for the success of large-language models and other nonlinear signal processing applications. The rationale for SDPA has been based upon "query, key, value" concepts borrowed from database theory, but these concepts are difficult to reconcile with standard methods in mathematical signal processing. We show that SDPA can be rewritten in a different but mathematically equivalent form as a projection of the input vectors onto a common surface determined by the inputs themselves. Therefore SDPA discovers nonlinear dependencies in the input that are time-dependent and context-dependent. The rewritten form of SDPA permits increased speed of both feedforward and learning algorithms, but more importantly suggests potential extensions. In the context of language, we re-interpret the role of SDPA as finding a time-dependent contextual meaning determined by the surface on which the set of input vectors lies. Input token embeddings are then modified by the local context surface. This interpretation differs substantially from the concept of "self-attention", and provides a strong justification for the use of SDPA for time-series data with time-varying local nonlinear dependencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。