揭示现代序列模型的数学本质：用非线性振荡器解释其计算机制
An explicit operator explains end-to-end computation in the modern neural networks used for sequence and language modeling
- 将状态空间模型映射为可解析求解的非线性振荡网络
- 推导出S4D前向传播的精确算子表达式，完整描述输入输出关系
- 通过波状活动和非线性解码揭示序列分类的物理机制
我们建立了状态空间模型（SSM）与一类可精确求解的非线性振荡器网络之间的数学对应关系。以结构化状态空间序列模型（S4）的对角线线性时不变实现（S4D）为例，该对应将S4D嵌入环形网络拓扑中，使近期输入以活动波的形式沿一维网络布局传播。我们推导出S4D完整前向传播的精确算子表达式，实现了对其全输入-输出映射的解析表征。该表达式表明，系统中的非线性解码器诱发了信息携带波间的相互作用，从而实现真实序列的分类。这些结果推广至现代各类SSM架构，表明它们均具有精确的数学描述及清晰的物理解释，为这类系统提供了新的可解释性视角。
原文摘要 · Abstract (English)
We establish a mathematical correspondence between state space models, a state-of-the-art architecture for capturing long-range dependencies in data, and an exactly solvable nonlinear oscillator network. As a specific example of this general correspondence, we analyze the diagonal linear time-invariant implementation of the Structured State Space Sequence model (S4). The correspondence embeds S4D, a specific implementation of S4, into a ring network topology, in which recent inputs are encoded, as waves of activity traveling over the one-dimensional spatial layout of the network. We then derive an exact operator expression for the full forward pass of S4D, yielding an analytical characterization of its complete input-output map. This expression reveals that the nonlinear decoder in the system induces interactions between these information-carrying waves that enable classifying real-world sequences. These results generalize across modern SSM architectures, and show that they admit an exact mathematical description with a clear physical interpretation. These insights enable a new level of interpretability for these systems in terms of nonlinear oscillator networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。