用复数编码同时表示内容和位置，提升注意力模型效率。
CoPE: A Lightweight Complex Positional Encoding
- 用复数嵌入分离语义与位置信息，实部表内容，虚部表位置。
- 在GLUE上优于RoPE等方法，计算开销更低，无长期衰减问题。
- 适合追求高效、低延迟的NLP模型部署场景。
近期研究证明了位置编码在Transformer架构中的有效性。通过引入位置信息，该方法为建模不同序列位置间依赖关系提供了关键指导。本文提出CoPE（轻量级复数位置编码），一种新架构，利用复数编码同时捕捉内容与位置信息。传统位置编码被替换为复数嵌入，其中实部表示语义内容，虚部表示位置信息。在Transformer第一层引入相位感知注意力以捕捉依赖位置的模式，后续层使用标准注意力。实验表明，CoPE无长期衰减现象，且与线性注意力兼容。在GLUE基准测试中，其性能优于RoPE、正弦及可学习位置编码，同时计算复杂度更低。
原文摘要 · Abstract (English)
Recent studies have demonstrated the effectiveness of position encoding in transformer architectures. By incorporating positional information, this approach provides essential guidance for modeling dependencies between elements across different sequence positions. We introduce CoPE (a lightweight Complex Positional Encoding), a novel architecture that leverages complex-valued encoding to encode both content and positional information. Our approach replaces traditional positional encodings with complex embeddings where the real part captures semantic content and the imaginary part encodes positional information. We introduce phase-aware attention in the first layer of the transformer model to capture position-dependent patterns, followed by standard attention layers for higher-levels. We show that CoPE doesn't exhibit long term decay and is compatible with linear attention. Experimental evaluation on the GLUE benchmark suggest that our approach achieves superior performance with less computational complexity, compared to RoPE, Sinusoidal and Learned positional encodings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。