用非线性查询提升注意力机制，效果优于传统线性投影。
Beyond Linearity in Attention Projections: The Case for Nonlinear Queries
- 将查询投影改为带残差的非线性结构，保留原有性能基线。
- 在小规模GPT模型上验证，对数损失降低2.40%，困惑度降6.81%。
- 适合关注注意力机制改进或模型轻量化的研究者阅读。
近期代数分析表明，在仅解码器或仅编码器的Transformer中，查询投影 $W_Q$ 可设为恒等矩阵而不会明显影响性能，因为注意力仅依赖于 $XW_Q, XW_K, XW_V$ 的乘积，允许基变换被相邻层吸收并传播。本文将 $W_Q \in \R^{d \times d}$ 替换为形如 $Q(X) = X + f_θ(X)$ 的非线性残差结构，其中 $f_θ$ 是参数量为 $d^2 + O(d)$ 的瓶颈MLP。恒等项为非线性提供了已知良好的先验锚点。在类似GPT-3小模型的实验中，该方法一致优于基线（验证对数损失降低2.40%，困惑度降低6.81%），且表现优于参数多12.5%的模型。这些结果推动了在更大规模和跨模态场景下的进一步研究。
原文摘要 · Abstract (English)
Recent algebraic analysis shows that in decoder-only and encoder-only transformers, the Query projection $W_Q$ may be set to identity without noticeable performance deterioration. This is possible because attention depends on $X$ only through the products $XW_Q, XW_K, XW_V$, allowing basis transformations to be absorbed by adjacent layers and propagated through the network. We replace $W_Q \in \R^{d \times d}$ with a nonlinear residual of the form $Q(X) = X + f_θ(X)$, where $f_θ$ is a bottleneck MLP with $d^2 + O(d)$ parameters. The identity term anchors the nonlinearity to a known-good prior. Experiments on GPT-3 small style models show consistent improvement over the baseline ($2.40\%$ lower validation log-loss, $6.81\%$ lower perplexity), comfortably outperforming a model with 12.5\% more non-embedding parameters. These results motivate investigation at larger scales and across modalities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。