用热带几何分析注意力的空间划分，揭示Transformer的几何容量极限。
Geometric Capacity of Transformers: A Tropical Geometry Perspective
- 引入热带几何建模注意力路由，将查询空间划分为多面体区域。
- 推导出Transformer线性区域数的紧致渐近界，为Θ(N^{min{H,d_model-1}L})。
- 适用于理解模型复杂度、设计高效架构的研究者。
为量化Transformer的几何容量，本文构建了一个热带几何框架，用于分析条件自注意力诱导的空间划分。在零温度极限下，固定键的top-1路由可精确表示为查询空间中的幂图，而辅助的对数提升值参数化则给出向量值热带有理表示。对于序列长度为N、注意力头数为H的多头自注意力（MHSA），联合路由几何由各头牛顿多面体的闵可夫斯基和编码，给出O(N^H)的通用上界，并在头数达到内在维度d_model时优化为O((HN)^{d_model-1})。进一步拓展至深度L，首次获得Transformer线性区域数的紧致渐近界：Θ(N^{min{H,d_model-1}L})。此外，有限温度softmax保持top-1路由结构，并在路由边界外呈现指数衰减的局部逼近与微分界。
原文摘要 · Abstract (English)
To quantify the geometric capacity of transformers, we develop a tropical-geometric framework for analyzing the spatial partitions induced by conditioned self-attention. In the zero-temperature limit, we show that fixed-key top-$1$ routing is exactly represented by a power diagram in query space, while an auxiliary log-lifted value parameterization yields a vector-valued tropical rational representation. For Multi-Head Self-Attention (MHSA) with sequence length $N$ and $H$ attention heads, the joint routing geometry is encoded by Minkowski sums of headwise Newton polytopes, giving an $\mathcal{O}(N^H)$ universal bound that sharpens to $\mathcal{O}((HN)^{d_{\mathrm{model}}-1})$ once the number of heads reaches the intrinsic dimension $d_{\mathrm{model}}$. Extending this analysis across depth $L$, we derive the first tight asymptotic bounds on the number of linear regions in transformers ($Θ\!\left(N^{\min\{H,d_{\mathrm{model}}-1\}L}\right)$). We further show that finite-temperature softmax preserves the top-$1$ routing structure and admits exponentially decaying local approximation and differential bounds away from routing boundaries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。