Transformer记忆能力极限可被理论证明，且与输入长度无关。
On the Optimal Memorization Capacity of Transformers
- 用自注意力机制实现高效序列记忆，参数仅需√N量级
- 序列到序列任务中需√(nN)参数，且此量级不可再少
- 适合研究模型记忆机制的算法与理论学者
近期机器学习研究关注Transformer的记忆容量，但其效率尚不明确。我们证明在下个词预测任务中，Transformer仅需˜O(√N)参数即可对N个长度为n的输入序列进行标签记忆，该结果在对数因子内为最优。这表明由于参数共享，记忆效率几乎不受输入长度n影响。在序列到序列设置中，˜O(√(nN))参数既充分也必要，至少对使用hardmax的Transformer成立。结果说明：自注意力能高效识别序列，但前馈网络在关联标签与每个词时成为瓶颈。
原文摘要 · Abstract (English)
Recent research in the field of machine learning has increasingly focused on the memorization capacity of Transformers, but how efficient they are is not yet well understood. We demonstrate that Transformers can memorize labels with $\tilde{O}(\sqrt{N})$ parameters in a next-token prediction setting for $N$ input sequences of length $n$, which is proved to be optimal up to logarithmic factors. This indicates that Transformers can efficiently perform memorization with little influence from the input length $n$ owing to the benefit of parameter sharing. We also analyze the memorization capacity in the sequence-to-sequence setting, and find that $\tilde{O}(\sqrt{nN})$ parameters are not only sufficient, but also necessary at least for Transformers with hardmax. These results suggest that while self-attention mechanisms can efficiently identify input sequences, the feed-forward network becomes a bottleneck when associating a label to each token.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。