用Lp范数改进Transformer的查询键归一化，提升训练稳定性
Enhanced QKNorm normalization for neural transformers with the Lp norm
- 提出基于Lp范数的QKNorm通用化方法，支持非欧几里得归一化
- 实验验证该方法在简单任务上有效，能稳定模型学习
- 适合关注注意力机制归一化设计的研究者
查询向量与键向量的归一化是Transformer架构中的关键环节，可确保学习过程不受向量尺度影响。现有多种归一化方法。本文提出一种QKNorm归一化方案的推广形式，基于Lp范数，使非欧几里得范数得以应用。初步实验表明，该方法在简单问题上表现良好，具备可行性。
原文摘要 · Abstract (English)
The normalization of query and key vectors is an essential part of the Transformer architecture. It ensures that learning is stable regardless of the scale of these vectors. Some normalization approaches are available. In this preliminary work, a generalization of the QKNorm normalization scheme is proposed. The approach is based on the Lp norm, allowing non-Euclidean norms to be employed. Experimental results demonstrate the suitability of the method for a simple problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。