揭示旋转位置编码的相位结构,统一分析语义连续性与执行边界控制。
Phase Structure in Rotary Attention: A Spectral Framework for Semantic Continuity and Execution-Boundary Governance
- 以隐藏状态序列为谱分解域,构建旋转注意力的相位分析框架。
- 证明相位偏移有界时,注意力分数衰减可控,保障模型稳定性。
- 区分表征连续性与执行合法性,适用于可解释性与模型治理研究。
Transformer语言模型通常通过向量几何分析,但有序上下文与旋转位置编码在查询-键交互中引入了明确的相位结构。本文提出一个有界谱框架,用于分析旋转相位对齐、隐藏状态连续性及语义漂移的影响,不将语言模型视为真实的物理波系统。首先识别隐藏状态序列而非词典索引为谱分解的有效域;其次将旋转位置嵌入(RoPE)注意力得分表示为加权余弦项之和,并证明局部稳定性引理:相位偏移有界可限制预软最大值得分的退化。为拓展相位分析至非原生RoPE坐标,论文定义复数模态坐标系,基于固定正交方向对,并引入加权相干函数描述隐藏状态轨迹。该框架严格区分表征连续性与执行边界可接受性:内部一致性可描述任务相关关系保持,但不能授权实质性转换。相较现有几何、谱、相位调制、表征分析与机制可解释性理论,本框架提供一套理论与方法论体系,界定何时谱结构可解释连续性,何时治理必须作为外部执行约束。
原文摘要 · Abstract (English)
Transformer language models are usually analyzed through vector geometry, yet ordered context and rotary position encoding introduce explicit phase structure into query-key interactions. This paper develops a bounded spectral framework for examining rotary phase alignment, hidden-state continuity, and semantic drift without treating language models as literal physical wave systems. It first identifies ordered hidden-state sequences, rather than vocabulary indices, as valid domains for spectral decomposition. It then derives the Rotary Position Embedding (RoPE) attention score as a sum of magnitude-weighted cosine terms and proves a local stability lemma: uniformly bounded phase displacement limits degradation of the corresponding pre-softmax score. To extend phase analysis beyond native RoPE coordinates, the paper defines complex modal coordinates over fixed orthonormal direction pairs and introduces a weighted coherence functional for hidden-state trajectories. These constructions support a strict distinction between representational continuity and execution-boundary admissibility. Internal coherence may describe preservation of task-relevant relations, but it cannot authorize a consequential transition. Positioned against existing geometric, spectral, phase-modulation, representation-analysis, and mechanistic-interpretability accounts, the framework contributes a theoretical and methodological program for determining when spectral structure explains continuity and when governance must remain an external predicate over execution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。