证明了Transformer中多头注意力机制存在高性能子网络。
The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms
- 在随机初始化的多头注意力中，通过理论推导证明强彩票票券存在。
- 当隐层维度达O(d log(Hd^{3/2}))时，近似任意目标MHA的概率很高。
- 首次将强彩票票券理论拓展至无归一化层的Transformer架构。
强彩票票券假说（SLTH）认为，高性能子网络（即强彩票票券，SLT）隐藏于随机初始化的神经网络中。尽管已有理论研究在多种网络架构中验证了该假说，但对Transformer架构，尤其是其核心组件——多头注意力（MHA）机制，仍缺乏理论支持。为此，本文首次对MHA中的SLT存在性进行理论分析。证明：若一个具有H个头、输入维度d的随机初始化MHA，其键和值的隐层维度为O(d log(Hd^{3/2}))，则以高概率包含一个能逼近任意同输入维度目标MHA的SLT。进一步基于此理论，将SLTH推广至不含归一化层的Transformer模型。实验验证表明，源模型（MHA与Transformer）中SLT与其目标近似之间的近似误差，随源模型隐层维度增加而呈指数下降。
原文摘要 · Abstract (English)
The strong lottery ticket hypothesis (SLTH) conjectures that high-performing subnetworks, called strong lottery tickets (SLTs), are hidden in randomly initialized neural networks. Although recent theoretical studies have established the SLTH across various neural architectures, the SLTH for transformer architectures still lacks theoretical understanding. In particular, the current theory of the SLTH does not yet account for the multi-head attention (MHA) mechanism, a core component of transformers. To address this gap, we introduce a theoretical analysis of the existence of SLTs within MHAs. We prove that, if a randomly initialized MHA of $H$ heads and input dimension $d$ has the hidden dimension $O(d\log(Hd^{3/2}))$ for the key and value, it contains an SLT that approximates an arbitrary MHA with the same input dimension with high probability. Furthermore, by leveraging this theory for MHAs, we extend the SLTH to transformers without normalization layers. We empirically validate our theoretical findings, demonstrating that the approximation error between the SLT within a source model (MHA and transformer) and an approximate target counterpart decreases exponentially by increasing the hidden dimension of the source model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。