用共享权重的自注意力机制，让BERT更省参数、更快训练且更抗干扰。
Does Self-Attention Need Separate Weights in Transformers?
- 三矩阵合一：键、值、查询共用一个权重矩阵，减少参数量。
- 参数量降66.53%，训练时间减至十分之一，小任务精度提升超5%。
- 适合追求轻量化、高泛化能力的NLP应用，尤其在噪声数据中表现优。
自注意力机制虽能捕捉长程依赖，但存在计算复杂度高及处理有方向性序列数据困难的问题。本文提出一种基于共享权重的自注意力BERT模型，仅学习一个权重矩阵用于键、值、查询表示，而非传统三个独立矩阵。该方法使注意力模块参数量减少66.53%，训练时间降低约十分之一。实验表明，在小规模任务上，该模型在GLUE基准上优于标准BERT基线，特别是在噪声数据和域外数据上展现出更强泛化能力，准确率分别提升0.38%、5.81%和1.06%。模型与源码已公开。
原文摘要 · Abstract (English)
The success of self-attention lies in its ability to capture long-range dependencies and enhance context understanding, but it is limited by its computational complexity and challenges in handling sequential data with inherent directionality. This work introduces a shared weight self-attention-based BERT model that only learns one weight matrix for (Key, Value, and Query) representations instead of three individual matrices for each of them. Our shared weight attention reduces the training parameter size by more than half and training time by around one-tenth. Furthermore, we demonstrate higher prediction accuracy on small tasks of GLUE over the BERT baseline and in particular a generalization power on noisy and out-of-domain data. Experimental results indicate that our shared self-attention method achieves a parameter size reduction of 66.53% in the attention block. In the GLUE dataset, the shared weight self-attention-based BERT model demonstrates accuracy improvements of 0.38%, 5.81%, and 1.06% over the standard, symmetric, and pairwise attention-based BERT models, respectively. The model and source code are available at Anonymous.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。