提出神经元级动态模型,让每个神经元独立演化,提升序列建模性能。
Topological Neural Dynamics: A Neuron-wise Framework for Sequence Modeling
- 将神经网络动态从层间同步改为神经元独立演化
- 在单人乒乓任务中实现17.47次连续击球,超最强基线三倍
- 适合需要精细动态建模的序列任务,如强化学习与行为克隆
现有序列模型(如RNN、LSTM、连续时间网络和Transformer)均采用层内统一动力学机制,所有神经元通过共享参数算子共同演化,限制了个体自由度。然而,在复杂系统中,全局行为常源于局部单元通过结构化连接的相互作用。受此启发,我们提出拓扑神经动力学(TND),将计算从层级动态转向神经元级动态。TND将神经网络视为有向神经元图、交互算子与局部动力学函数的组合,每个神经元独立演化,集体计算由显式图拓扑中的交互涌现。我们以离散时间图耦合动力系统形式实现TND,并在单人乒乓行为克隆任务中进行验证。相比普通RNN、稀疏RNN、LSTM、闭式连续时间神经网络(CfC)和Transformer基线,TND取得最佳击球成功率,平均每轮连续击球17.47次,超过最强基线三倍。结果表明,从层内到神经元级动力学的转变,为序列建模提供了有效的归纳偏置。
原文摘要 · Abstract (English)
Existing sequence models, including RNNs, LSTMs, continuous-time networks, and Transformers, share a common structural principle: layer-wise dynamics, where all neurons in the same layer co-evolve through a shared parameterized operator, leaving individual neurons no freedom to evolve independently. Yet in many complex dynamical systems, rich global behavior emerges precisely from locally evolving units interacting through structured connectivity. Inspired by this principle, we introduce Topological Neural Dynamics (TND), a sequence modeling framework that shifts computation from layer-wise to neuron-wise dynamics. TND represents a neural system as a directed neuron graph, an interaction operator, and a local dynamics function, where each neuron evolves independently and collective computation emerges from interactions through the explicit graph topology. We instantiate TND as a discrete-time graph-coupled dynamical system and evaluate it as a case study on a behavior cloning task in single-player Pong. Compared with Vanilla RNN, Sparse RNN, LSTM, Closed-form continuous-time neural network (CfC), and Transformer baselines, TND achieves the best catch rate and a mean of 17.47 consecutive catches per round, more than three times that of the strongest baseline. These results suggest that shifting from layer-wise to neuron-wise dynamics provides an effective inductive bias for sequence modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。