通过共享基矩阵优化注意力机制，提升长序列建模效率与精度。
Shared DIFF Transformer
- 引入共享基矩阵与低秩更新，减少参数冗余。
- 在长序列任务中性能优于原DIFF Transformer，噪声抑制更强。
- 适合需要高效高精度注意力的NLP应用，如信息检索。
DIFF Transformer通过差异注意力机制增强对相关上下文的关注并抑制噪声，但其独立信号生成导致参数冗余和信息利用不充分。本文提出共享差异注意力变换器（Shared DIFF Transformer），借鉴差分放大器思想，引入共享基矩阵建模全局模式，并结合低秩更新提升任务特异性灵活性。该设计显著降低参数冗余，提高效率，同时保持强噪声抑制能力。实验表明，在长序列建模、关键信息提取和上下文学习等任务中，相比DIFF Transformer，本方法表现更优。工作为优化差异注意力机制提供了新颖高效的路径，推动了鲁棒Transformer架构的发展。
原文摘要 · Abstract (English)
DIFF Transformer improves attention allocation by enhancing focus on relevant context while suppressing noise. It introduces a differential attention mechanism that calculates the difference between two independently generated attention distributions, effectively reducing noise and promoting sparse attention patterns. However, the independent signal generation in DIFF Transformer results in parameter redundancy and suboptimal utilization of information. In this work, we propose Shared DIFF Transformer, which draws on the idea of a differential amplifier by introducing a shared base matrix to model global patterns and incorporating low-rank updates to enhance task-specific flexibility. This design significantly reduces parameter redundancy, improves efficiency, and retains strong noise suppression capabilities. Experimental results show that, compared to DIFF Transformer, our method achieves better performance in tasks such as long-sequence modeling, key information retrieval, and in-context learning. Our work provides a novel and efficient approach to optimizing differential attention mechanisms and advancing robust Transformer architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。