通过物理动量机制，实现单层上下文学习与频谱可解释性。
Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability
- 引入动量注意力，用物理动力学建模查询键值的动态变化。
- 单层即可完成归纳推理，频谱分析显示信号在频域正交分离。
- 适合研究生成模型机制、物理启发神经网络的学者使用。
机制可解释性（MI）将Transformer视为精确的计算图。本文将其扩展为具有守恒律和时变交流动力学的物理电路，提出动量注意力：通过动量差算子 $p_t = q_t - q_{t-1}$ 嵌入物理先验，对查询和键实施辛变换 $ ilde{q}_t = q_t + γp_t$。我们发现核心对偶性——物理剪切等价于高通滤波。该对偶性突破了归纳头形成需至少两层（$L \geq 2$）的拓扑限制：因直接获取速度信息，实现单层归纳与基于巴德图的频谱取证。我们证明正交性定理：当低通旋转位置编码（RoPE）与高通动量结合时，直流（语义）与交流（机制）信号分属正交频带。5,100+受控实验（附录A–R及27个Jupyter笔记本验证）表明，125M动量模型在归纳密集任务上超越预期，验证损失仅比350M基线高约2.9%。专用关联回忆实验揭示缩放律 $γ^* = 4.17 \times N^{-0.74}$，确立动量与深度的可替代性。本框架为生成式AI、哈密顿物理与信号处理提供互补分析工具。
原文摘要 · Abstract (English)
The Mechanistic Interpretability (MI) program has mapped the Transformer as a precise computational graph. We extend this graph with a conservation law and time-varying AC dynamics, viewing it as a physical circuit. We introduce Momentum Attention, a symplectic augmentation embedding physical priors via the kinematic difference operator $p_t = q_t - q_{t-1}$, implementing the symplectic shear $\hat{q}_t = q_t + γp_t$ on queries and keys. We identify a fundamental Symplectic-Filter Duality: the physical shear is mathematically equivalent to a High-Pass Filter. This duality is our cornerstone contribution -- by injecting kinematic momentum, we sidestep the topological depth constraint ($L \geq 2$) for induction head formation. While standard architectures require two layers for induction from static positions, our extension grants direct access to velocity, enabling Single-Layer Induction and Spectral Forensics via Bode Plots. We formalize an Orthogonality Theorem proving that DC (semantic) and AC (mechanistic) signals segregate into orthogonal frequency bands when Low-Pass RoPE interacts with High-Pass Momentum. Validated through 5,100+ controlled experiments (documented in Supplementary Appendices A--R and 27 Jupyter notebooks), our 125M Momentum model exceeds expectations on induction-heavy tasks while tracking a 350M baseline within $\sim$2.9% validation loss. Dedicated associative recall experiments reveal a scaling law $γ^* = 4.17 \times N^{-0.74}$ establishing momentum-depth fungibility. We offer this framework as a complementary analytical toolkit connecting Generative AI, Hamiltonian Physics, and Signal Processing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。