通过三相结构提升Transformer的稳定性与效率，实现更优的生成性能。
Three-Phase Transformer
- 将隐藏向量分三通道，用旋转和归一化保持相位一致
- 在123M参数下比基准模型降低7.20%困惑度，提速1.93倍
- 适合追求模型稳定性和推理效率的研究者与工程师
我们提出Three-Phase Transformer（3PT），一种基于标准SwiGLU + RMSNorm + RoPE + GQA架构的解码器仅用变压器残差流结构先验。隐藏向量被划分为N个等长循环通道，每个通道由相位尊重操作维护：注意力与前馈层间进行2D吉文斯旋转，每通道旋转角度为θ + i*(2π/N)，并施加头数约束使GQA头与划分对齐。该架构在打乱与重建之间形成自稳态平衡，非外挂模块。划分出一个与通道正交的一维直流子空间，注入固定Gabriel's horn轮廓r(p) = 1/(p+1)作为绝对位置侧通道，与RoPE的相对位置旋转正交组合。标准N=3借鉴三相交流电概念，三个相差120度的正弦波叠加为零且无反相关对。在WikiText-103上123M参数时，3PT相较匹配的RoPE-only基线在增加1,536参数（总参数的0.00124%）条件下，实现-7.20%困惑度（-2.62%比特/字节），步数收敛速度提升1.93倍（壁钟时间提升1.64倍）。N作为参数共享调节旋钮而非唯一最优值：在5.5M参数下，对{1,2,3,4,6,8,12}的扫面近似单调，N=1最优；在123M参数下三种子扫面显示N=3与N=1统计上无差异。核心机制包括通道分区残差流、块内旋转、相位归一化及圆锥直流注入。我们分析了(1)无需显式强制的几何自稳态，是神经网络守恒律框架的新实例；(2)12层深度下旋转角漂移呈U型分布；(3)与RoPE、注意力和前馈网络的正交组合。
原文摘要 · Abstract (English)
We present Three-Phase Transformer (3PT), a residual-stream structural prior for decoder-only Transformers on a standard SwiGLU + RMSNorm + RoPE + GQA backbone. The hidden vector is partitioned into N equally-sized cyclic channels, each maintained by phase-respecting ops: a per-channel RMSNorm, a 2D Givens rotation between attention and FFN that rotates each channel by theta + i*(2*pi/N), and a head-count constraint aligning GQA heads with the partition. The architecture is a self-stabilizing equilibrium between scrambling and re-imposition, not a bolted-on module. The partition carves out a one-dimensional DC subspace orthogonal to the channels, into which we inject a fixed Gabriel's horn profile r(p) = 1/(p+1) as an absolute-position side-channel composing orthogonally with RoPE's relative-position rotation. The canonical N=3 borrows its metaphor from balanced three-phase AC, where three sinusoids 120 degrees apart sum to zero with no anti-correlated pair. At 123M parameters on WikiText-103, 3PT achieves -7.20% perplexity (-2.62% bits-per-byte) over a matched RoPE-Only baseline at +1,536 parameters (0.00124% of total), with 1.93x step-count convergence speedup (1.64x wall-clock). N behaves as a parameter-sharing knob rather than a unique optimum: at 5.5M an N-sweep over {1,2,3,4,6,8,12} is near-monotone with N=1 winning; at 123M a three-seed sweep finds N=3 and N=1 statistically indistinguishable. The load-bearing mechanism is the channel-partitioned residual stream, per-block rotation, per-phase normalization, and horn DC injection. We characterize (a) self-stabilization of the geometry without explicit enforcement, a novel instance of the conservation-law framework for neural networks; (b) a U-shaped depth profile of rotation-angle drift at 12 layers; (c) orthogonal composition with RoPE, attention, and FFN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。