用几何原理让Transformer自动分离语义与语法，提升可解释性
PRISM: Deriving a White-Box Transformer as a Signal-Noise Decomposition Operator via Maximum Coding Rate Reduction
- 基于最大编码率压缩原理设计白盒注意力机制
- 模型自发分出高低频头，分别处理长程语义和局部语法
- 适合关注模型可解释性与架构理论的AI研究者
深度学习模型，尤其是Transformer，常被批评为“黑箱”，缺乏可解释性。我们提出Prism，一种基于最大化编码率压缩（MCR²）原则构建的白盒注意力架构。通过将注意力机制建模为在信号-噪声流形上的梯度上升过程，引入特定的无理频率分离（π-RoPE），强制语义（信号）与句法（噪声）子空间的非相干性。实证表明，这些几何归纳偏置可单独引发无监督功能解耦。Prism自发地将注意力头分化为频谱不同的模式：低频头捕捉长程因果依赖（信号），高频头处理局部句法约束与结构伪影。为解释这一频谱现象，我们建立注意力机制与哈密顿动力系统的类比，发现标准旋转位置编码（RoPE）的几何级数会诱导密集共振网络（阿诺德舌），导致特征秩坍缩。在124M参数模型上训练于OpenWebText的实验证明，Prism自发隔离了注意力汇聚病灶，并维持层间等熵信息流动。进一步提出物理启发的即插即用干预KAM-RoPE，适用于大语言模型。结果表明，通过严谨的几何构造，可统一可解释性与性能，提供对启发式架构修改的理论替代方案。
原文摘要 · Abstract (English)
Deep learning models, particularly Transformers, are often criticized as "black boxes" and lack interpretability. We propose Prism, a white-box attention-based architecture derived from the principles of Maximizing Coding Rate Reduction ($\text{MCR}^2$). By modeling the attention mechanism as a gradient ascent process on a distinct signal-noise manifold, we introduce a specific irrational frequency separation ($π$-RoPE) to enforce incoherence between signal (semantic) and noise (syntactic) subspaces. We show empirical evidence that these geometric inductive biases can induce unsupervised functional disentanglement alone. Prism spontaneously specializes its attention heads into spectrally distinct regimes: low-frequency heads capturing long-range causal dependencies (signal) and high-frequency heads handling local syntactic constraints and structural artifacts. To provide a theoretical grounding for these spectral phenomena, we draw an analogy between attention mechanism and a Hamiltonian dynamical system and identify that the standard geometric progression of Rotary Positional Embeddings (RoPE) induces dense resonance networks (Arnold Tongues), leading to feature rank collapse. Empirical validation on 124M-parameter models trained on OpenWebText demonstrates that Prism spontaneously isolates the Attention Sink pathology and maintains isentropic information flow across layers. Further, we suggest a physics-informed plug-and-play intervention KAM-RoPE for large language models (LLMs). Our results suggest that interpretability and performance can be unified through principled geometric construction, offering a theoretically grounded alternative to heuristic architectural modifications
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。