重新发现循环网络中双线性状态更新的计算价值
Revisiting Bi-Linear State Transitions in Recurrent Neural Networks
- 将隐藏单元视为计算参与者而非仅存记忆
- 双线性操作在状态追踪任务中具天然归纳偏置
- 揭示了Mamba等线性模型在复杂度层级中的位置
循环神经网络中隐藏单元的传统角色是建模记忆，研究多聚焦于通过门控机制增强信息保留。本文从另一视角出发，认为隐藏单元应主动参与网络计算。我们重新审视双线性操作——即隐藏状态与输入嵌入间的乘积交互——并从理论与实证两方面证明，这类操作构成状态追踪任务中隐藏状态演化的自然归纳偏置。此类任务是最基础的需隐藏单元主动贡献行为的任务。我们进一步发现，双线性状态更新形成一个自然层次结构，对应状态追踪任务复杂度递增；诸如Mamba等流行的线性循环网络位于该层级的最低复杂度中心。
原文摘要 · Abstract (English)
The role of hidden units in recurrent neural networks is typically seen as modeling memory, with research focusing on enhancing information retention through gating mechanisms. A less explored perspective views hidden units as active participants in the computation performed by the network, rather than passive memory stores. In this work, we revisit bilinear operations, which involve multiplicative interactions between hidden units and input embeddings. We demonstrate theoretically and empirically that they constitute a natural inductive bias for representing the evolution of hidden states in state tracking tasks. These are the simplest type of tasks that require hidden units to actively contribute to the behavior of the network. We also show that bilinear state updates form a natural hierarchy corresponding to state tracking tasks of increasing complexity, with popular linear recurrent networks such as Mamba residing at the lowest-complexity center of that hierarchy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。