从线性代数角度分析注意力层的记忆能力，揭示其容量与数据库规模的关系。
Paying Attention to Facts: Quantifying the Knowledge Capacity of Attention Layers
- 用三阶张量衡量数据库规模，提出注意力层秩与数据库秩的关联性
- 实验显示注意力层秩与数据库秩存在可验证的对应关系
- 发现查询-键权重和softmax对记忆容量的关键影响，适合模型优化研究者
本文从线性代数视角研究单层仅含注意力机制的Transformer（即注意力层）对数据库中事实的记忆能力。为每个数据库构建一个三阶张量，并将其张量秩作为数据库规模的度量，给出了该秩与数据库属性的关系边界。同时定义了对应注意力层的三阶张量，通过一组玩具模型与随机数据库的实验，实证展示了其秩与数据库秩之间的关系。研究揭示了值输出权重、查询-键权重的作用，以及argmax和softmax对秩的影响，阐明了Transformer中事实召回的‘加法模式’，并提出在不增加参数量的前提下提升层容量的可能性。
原文摘要 · Abstract (English)
In this paper, we investigate the ability of single-layer attention-only transformers (i.e. attention layers) to memorize facts contained in databases from a linear-algebraic perspective. We associate with each database a 3-tensor, propose the rank of this tensor as a measure of the size of the database, and provide bounds on the rank in terms of properties of the database. We also define a 3-tensor corresponding to an attention layer, and empirically demonstrate the relationship between its rank and database rank on a dataset of toy models and random databases. By highlighting the roles played by the value-output and query-key weights, and the effects of argmax and softmax on rank, our results shed light on the `additive motif' of factual recall in transformers, while also suggesting a way of increasing layer capacity without increasing the number of parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。