arXiv:2510.06662cs.LGstat.ML2025-10中稿 · ICLR被引 4

头数少时模型需海量参数才能逼近,多头则高效表达。

The Effect of Attention Head Count on Transformer Approximation

  • 用泛化检索任务建模,分析注意力头数对表达能力的影响。
  • 头数不足时参数量需随ε以指数级增长,头数够多则可高效逼近。
  • 单头模型靠嵌入维度记忆输入,前馈层完成全部近似任务。

Transformer已成为序列建模的主流架构,但其结构参数如何影响表达能力仍不清晰。本文研究Transformer的逼近性质,重点关注注意力头数的作用。我们提出一个广义的$D$-检索任务,证明其在连续函数空间中稠密,为理论框架奠定基础。进一步建立了$ε$-逼近所需的参数复杂度上下界:当注意力头数足够多时,可实现高效逼近;而头数过少时,参数量至少需达到$O(1/ε^{cT})$,其中$ c $为常数,$ T $为序列长度。据我们所知,这是首个在非线性且实际相关场景下的严格下界。我们还分析了单头情形,发现嵌入维度为$O(T)$时,可通过前馈块完全记忆输入,实现无误差逼近。实验在合成数据和真实任务上验证了理论结果的实用性。

原文摘要 · Abstract (English)

Transformer has become the dominant architecture for sequence modeling, yet a detailed understanding of how its structural parameters influence expressive power remains limited. In this work, we study the approximation properties of transformers, with particular emphasis on the role of the number of attention heads. Our analysis begins with the introduction of a generalized $D$-retrieval task, which we prove to be dense in the space of continuous functions, thereby providing the basis for our theoretical framework. We then establish both upper and lower bounds on the parameter complexity required for $ε$-approximation. Specifically, we show that transformers with sufficiently many heads admit efficient approximation, whereas with too few heads, the number of parameters must scale at least as $O(1/ε^{cT})$, for some constant $c$ and sequence length $T$. To the best of our knowledge, this constitutes the first rigorous lower bound of this type in a nonlinear and practically relevant setting. We further examine the single-head case and demonstrate that an embedding dimension of order $O(T)$ allows complete memorization of the input, where approximation is entirely achieved by the feed-forward block. Finally, we validate our theoretical findings with experiments on both synthetic data and real-world tasks, illustrating the practical relevance of our results.

Transformer注意力机制理论分析逼近能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。