不同架构的神经网络竟有相似的慢速动态规律，揭示了深层共性。
Infrared Universality of Collective Dynamics across Transformer and State-Space Architectures
- 通过分析Mamba与Transformer的集体动力学，发现其红外区域存在相似的慢模式组织。
- 长序列下Mamba的幂律指数稳定在β≈-0.17，对应接近1/t的长期记忆行为。
- 即使微观机制不同，两类模型仍表现出近临界慢动态，适用于研究认知动力学。
不同神经网络架构是否发展出共同的集体动力学仍是未解问题。近期对Transformer语言模型的分析发现，其时间尺度态密度（TDOS）接近平坦且弱红外增强，与近临界长时记忆动态相关。本文检验Mamba模型中是否存在类似组织，其选择性状态空间动态提供了根本不同的微观机制。Mamba可从三个层次解析弛豫动力学：学习到的状态空间生成器本征谱、输入条件下的选择性重缩放，以及完整模块的雅可比矩阵测量的集体TDOS。三者不完全相同：选择性动态和剩余块变换显著重构了微观弛豫层级。然而，完整模块仍发展出可重复的慢模连续体，其红外部分随序列长度增加而逐步更清晰。累积分析得ρ(λ)∼λ^β，长序列下Mamba的指数稳定在βₘ≃-0.17。对应的记忆动力学为K(t)∼t^(-(1+β))，接近临界1/t regime。尽管微观机制迥异，Transformer全块谱也呈现相近的红外组织，代表性指数约为βₜᵣ∼-0.1。结果分离了显式状态空间记忆与集体红外组织，表明不同序列架构可发展出紧密相关的近临界慢模动态。该发现将红外集体组织扩展至Transformer之外，并为认知场理论描述的动力学结构提供了独立验证。
原文摘要 · Abstract (English)
Whether distinct neural architectures develop common collective dynamics remains an open question. Recent analysis of Transformer language models revealed a nearly flat, weakly infrared-enhanced time-scale density of states (TDOS) associated with near-marginal long-memory dynamics. Here we test whether a closely related organization emerges in Mamba, whose selective state-space dynamics provides a fundamentally different microscopic mechanism. Mamba allows relaxation dynamics to be resolved at three levels: the intrinsic spectrum of the learned state-space generator, its input-conditioned selective rescaling, and the collective TDOS of the complete block measured from its Jacobian. These spectra are not identical: selective dynamics and the remaining block transformations substantially reorganize the microscopic relaxation hierarchy. Nevertheless, the full block develops a reproducible slow-mode continuum whose infrared sector becomes progressively better resolved with increasing sequence length. Cumulative analysis yields $ρ(λ)\simλ^β$, with the long-sequence Mamba exponent stabilizing near $β_{\rm M}\simeq-0.17$. The corresponding memory dynamics follows $K(t)\sim t^{-(1+β)}$, close to the marginal $1/t$ regime. Despite fundamentally different microscopic dynamics, Transformer full-block spectra exhibit closely related infrared organization, with representative exponents of order $β_{\rm Tr}\sim-0.1$. These results separate explicit state-space memory from collective infrared organization and show that distinct sequence architectures can develop closely related near-marginal slow-mode dynamics. They extend infrared collective organization beyond Transformers and provide an independent test of the dynamical structure described by Cognitive Field Theory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。