改进旋转位置编码，节省近一半注意力参数。
CRoPE: Efficient Parametrization of Rotary Positional Embedding
- 用复数线性变换替代原实现，更自然且减少参数。
- 实验表明删减冗余参数对性能影响极小。
- 适合追求高效模型压缩与可解释性的研究者。
旋转位置编码已成为Transformer模型中编码位置信息的最先进方法。尽管其常以复数线性代数形式简洁表达，但我们发现实际的$Q/K/V$投影并不等价于复数线性变换。我们认为复数线性变换是更自然的参数化方式，能在注意力模块中节省近50%的参数。实证结果表明，去除这种冗余对模型性能几乎无影响。该改进实现了更高效的参数利用，并提供了更清晰的表征空间解释。
原文摘要 · Abstract (English)
Rotary positional embedding has become the state-of-the-art approach to encode position information in transformer-based models. While it is often succinctly expressed in complex linear algebra, we note that the actual implementation of $Q/K/V$-projections is not equivalent to a complex linear transformation. We argue that complex linear transformation is a more natural parametrization and saves near 50\% parameters within the attention block. We show empirically that removing such redundancy has negligible impact on the model performance. Our modification achieves more efficient parameter usage, as well as a cleaner interpretation of the representation space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。