将门控MLP看作破坏对称性的低秩双线性注意力，解释其有效性。
Gated MLPs as Symmetry-Broken Rank-1 Bilinear Attention
- 门控MLP本质是双线性注意力的低秩近似
- 非线性位置改变使两因子不对称，打破反向缩放对称性
- 为设计新架构提供理论启发，适合关注模型机制的读者
我们表明，传统的门控MLP可被视为一种双线性注意力机制的秩-1近似，其中包含两个分别对应查询和键的不同因子。进一步地，我们将非线性施加于其中一个因子时，会破坏这两个因子之间的交换对称性，对于非齐次激活函数，还会破坏逆缩放对称性。这一视角有助于解释门控MLP在实践中为何有效，并可为未来模型架构的设计提供指导。
原文摘要 · Abstract (English)
We show that the conventional gated MLP can be viewed as a rank-1 approximation to a bilinear attention mechanism with two distinct factors corresponding to the query and the key. We further show that moving the nonlinearity onto one factor breaks the exchange symmetry between the two factors and, for non-homogeneous activations, the inverse-scaling symmetry as well. This perspective may help explain why gated MLPs are effective in practice and inform the design of future architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。