揭示了softmax注意力的复杂度由支撑几何决定，提出精确的秩边界公式。
The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension
- 通过几何分析建立注意力秩的精确上界，区分支撑与交互几何影响。
- 球面自注意力秩为Θ(min{n,(1+β)^(d-1)/2})，全球几何下可达Θ(β^d/2)。
- 适用于理解大模型注意力机制、优化与理论分析的研究者。
什么几何控制归一化softmax注意力的秩复杂度？我们研究最大行ℓ₁近似秩，即保持所有有界向量输出的最小无约束秩。两个尖锐的最坏情况法则揭示了支撑几何的作用：固定d和误差ε，球面自注意力的秩为Θ_{d,ε}(min{n,(1+β)^{(d-1)/2}})，而全球几何增加一个径向自由度，在β≥β₀(d,ε)且n≥C_d e^{β/8}时，给出Θ_{d,ε}(β^{d/2})。对于单个头，行softmax消除了行标量逻辑方向：剩余可见查询-键交互维度r带来r/2的每实例上界，有界构造证明该指数为极小极大最优。近似交互子空间引入显式残差输出误差，并产生容差索引的SVD维度。在84头BERT-base校准集上，观察到多种头-温度设置下的适度有效维度下降，且与有限构造秩上界呈正相关。这些结果将支撑几何（决定最坏情况温度缩放）与softmax可见交互几何（控制单头近似复杂度）区分开来。
原文摘要 · Abstract (English)
Which geometry controls the rank complexity of normalized softmax attention? We study maximum-row-$\ell_1$ approximation rank, exactly the least unrestricted rank preserving every bounded vector-valued output. Two sharp worst-case laws isolate support geometry: for fixed $d$ and error $\varepsilon$, spherical self-attention has rank $Θ_{d,\varepsilon}(\min\{n,(1+β)^{(d-1)/2}\})$, while full-ball geometry adds one radial degree and, for $β\geβ_0(d,\varepsilon)$ and $n\ge C_d e^{β/8}$, gives $Θ_{d,\varepsilon}(β^{d/2})$. For a fixed head, row-softmax quotients out row-scalar logit directions: the remaining visible query--key interaction dimension $r$ yields an $r/2$ per-instance upper law, and bounded constructions show this exponent is minimax sharp. Approximate interaction subspaces incur an explicit residual output error and yield a tolerance-indexed SVD dimension. On an 84-head BERT-base calibration set, we observe modest effective-dimension reductions across many head--temperature settings, together with positive associations with finite constructive rank upper certificates. Together, these results separate support geometry, which sets worst-case temperature scaling, from softmax-visible interaction geometry, which controls per-head approximation complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。