arXiv:2607.10923cs.LG2026-07被引 2

揭示了Transformer模型中慢速模式的红外组织规律及其对记忆与认知的关键影响。

Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics

论文配图:Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics
图 1 · 摘自论文原文
  • 通过层雅可比矩阵分析,量化了训练过程中集体松弛谱的红外重分布机制。
  • 发现谱密度呈弱红外增强态,记忆核呈现近1/t的长时记忆标度行为。
  • 该规律在不同训练阶段、提示和模型规模下均稳定出现,适合研究大模型动力学者参考。

大型语言模型表现出显著的涌现行为,但其集体动态的物理机制仍不清晰。认知场理论预测,学习会重塑集体松弛谱,从而改变记忆自能、长记忆动力学和集体敏感性,这依赖于慢松弛模式的红外组织。本文直接在Transformer动态中检验该框架。利用公开的Pythia语言模型,从各层雅可比矩阵中提取松弛谱,覆盖训练过程、提示集合、网络深度和模型规模,实现了对认知场理论集体可观测量的定量测量。结果揭示明显的红外重分布:学习显著转移谱权重,同时保持近乎平坦但弱红外增强的时间尺度态密度 ρ(λ)∼λ^β,其中 β≈−0.1;相应记忆核表现出接近 K(t)∼1/t 的鲁棒长记忆标度。集体可观测量进一步显示临界形成过程:记忆自能在早期训练中达瞬时峰值后趋于亚稳态近临界态。提示分辨与词元子空间测量表明,不同局部雅可比矩阵均恢复共有的宏观总态密度,具有共享红外标度,符合粗粒化下的红外固定点组织。该红外组织在训练、提示集合、网络深度和模型规模下高度可重复,支持红外慢模组织是Transformer动态的稳健集体原则,并为认知场理论预测的集体可观测量提供了定量实验实现。

原文摘要 · Abstract (English)

Large language models exhibit remarkable emergent behaviors, yet the physical mechanism governing their collective dynamics remains poorly understood. Cognitive Field Theory predicts that learning reorganizes the collective relaxation spectrum, thereby modifying memory self-energy, long-memory dynamics, and collective susceptibility through the infrared organization of slow relaxation modes. Here we test this framework directly in Transformer dynamics. Using publicly available Pythia language models, we extract relaxation spectra from layer Jacobians throughout training, prompt ensembles, network depth, and model scale, allowing the collective observables of Cognitive Field Theory to be measured quantitatively. The measurements reveal pronounced infrared reorganization of the relaxation spectrum. Learning substantially redistributes spectral weight while preserving a nearly flat but weakly infrared-enhanced time-scale density of states, \( ρ(λ)\simλ^β, \qquad β\simeq-0.1, \) with a corresponding memory kernel exhibiting robust long-memory scaling close to \( K(t)\sim\frac{1}{t}. \) The collective observables further reveal a critical formation process: the memory self-energy reaches a transient maximum during early training before relaxing toward a metastable near-critical regime. Prompt-resolved and token-subspace measurements show that distinct local Jacobians recover a common macroscopic TDOS with shared infrared scaling, consistent with infrared fixed-point organization under coarse graining. The reproducibility of this infrared organization across training, prompt ensembles, network depth, and Transformer model scales supports infrared slow-mode organization as a robust collective principle of Transformer dynamics, providing a quantitative experimental realization of the collective observables predicted by Cognitive Field Theory.

Transformer动力学认知场长记忆

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。