用矩阵记忆取代残差流，让Transformer更高效
Residual Matrix Transformers: Scaling the Size of the Residual Stream
- 用外积记忆矩阵替代传统残差流，实现信息存储与读取
- 相同性能下减少58%算力、25%参数、41%训练数据
- 理论分析显示其更适合大规模扩展，适合追求效率的研究者
残差流在Transformer中充当特征存储与访问的内存总线（Elhage et al., 2021）。本文提出将残差流替换为外积记忆矩阵（Kohonen, 1972；Anderson, 1972），构建残差矩阵Transformer（RMT）。实验表明：1）残差流规模可独立于计算量和模型大小扩展，提升性能；2）RMT在相同损失下仅需58%的FLOPS、25%的参数和41%的训练数据；3）在下游任务上表现优于标准Transformer。理论分析揭示，RMT具备更优的残差流扩展性与方差传播特性。代码开源地址：https://github.com/bmac3/residual-matrix-transformer。
原文摘要 · Abstract (English)
The residual stream acts as a memory bus where transformer layers both store and access features (Elhage et al., 2021). We consider changing the mechanism for retrieving and storing information in the residual stream, and replace the residual stream of the transformer with an outer product memory matrix (Kohonen, 1972, Anderson, 1972). We call this model the Residual Matrix Transformer (RMT). We find that the RMT enjoys a number of attractive properties: 1) the size of the residual stream can be scaled independently of compute and model size, improving performance, 2) the RMT can achieve the same loss as the transformer with 58% fewer FLOPS, 25% fewer parameters, and 41% fewer training tokens tokens, and 3) the RMT outperforms the transformer on downstream evaluations. We theoretically analyze the transformer and the RMT, and show that the RMT allows for more efficient scaling of the residual stream, as well as improved variance propagation properties. Code for this project can be found at https://github.com/bmac3/residual-matrix-transformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。