让深层注意力值向量不依赖上下文，能提升模型性能且节省资源。
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
- 深层注意力使用与上下文无关的值向量，保留原始词元信息。
- 在780M模型上，21个基准平均得分匹配最优方法，但计算和内存更少。
- 提出价值库（BoV）机制，以稀疏参数存储词元特定值向量。
Transformer的成功很大程度归功于注意力层的设计。标准注意力层以残差流为输入，生成依赖上下文的查询、键和值向量。但我们发现，当深层网络仅学习与上下文无关的值向量时，模型性能显著提升，从而更好保留原始词元信息，而不依赖残差流中的上下文。此时，再加入依赖上下文的成分对整体基准性能提升有限。这些与上下文无关的值向量可作为稀疏模型参数存储，无需重新计算或持续缓存。通过系统消融实验，我们提出价值库（Bank of Values, BoV），一种新的注意力值向量计算方式：在最后三分之一层中为每个词元学习一个特定值向量的查找表。在135M和780M模型上，BoV均优于标准注意力，780M模型在21个基准上的平均得分达到当前最佳水平，且所需计算与内存更少。
原文摘要 · Abstract (English)
The success of the transformer architecture is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn only a context-free value vector to preserve the original token information, without drawing on any context from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component provides little additional benefit for aggregate benchmark performance. Such context-free value vectors can be stored as sparse model parameters, eliminating the need to recompute or persistently cache these values. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in attention by learning a lookup table of token-specific value vectors for each of the last third of layers. Across 135M and 780M models, BoV improves validation loss over standard attention and, at 780M, the average score across 21 benchmarks, matching the previous best method that adds token information to the value vector with less compute and memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。