解释深度网络权重为何具几何连续性，关键在残差连接与非线性对称性破缺。
Why Geometric Continuity Emerges in Deep Neural Networks: Residual Connections and Rotational Symmetry Breaking
- 残差连接使跨层梯度一致，对齐权重更新方向。
- 非线性激活打破旋转对称性，防止权重结构漂移。
- 适合研究模型内部动态与结构稳定性的读者。
深度网络中的权重重矩阵表现出几何连续性——相邻层的主奇异向量方向相近。尽管这一现象广泛被观察到，其成因仍不清楚。通过在小型MLP和小规模Transformer上的实验，我们发现两个机制：残差连接带来跨层梯度一致性，使权重更新方向对齐；非线性激活引起的对称性破缺，将所有层约束在共同坐标系中，避免了旋转漂移导致的结构不稳。关键发现是：虽非线性必要，但仅保持旋转不变的激活无法维持连续性，说明对称性破缺才是核心因素。激活函数集中连续性于首奇异方向，而归一化则将其分布至多个方向。在Transformer中，连续性具有投影特异性：读取残差流的Q、K、Gate、Up层在输入空间（α₁）展现连续性；写入残差流的O和Down层在输出空间（β₁）展现连续性；仅有V层因缺乏邻近非线性，连续性较弱。
原文摘要 · Abstract (English)
Weight matrices in deep networks exhibit geometric continuity -- principal singular vectors of adjacent layers point in similar directions. While this property has been widely observed, its origin remains unexplained. Through experiments on toy MLPs and small transformers, we identify two mechanisms: residual connections create cross-layer gradient coherence that aligns weight updates across layers, and symmetry-breaking nonlinearities constrain all layers to a shared coordinate frame, preventing the rotation drift that would otherwise destabilize weight structure. Crucially, a nonlinear but rotation-preserving activation fails to retain continuity, isolating symmetry breaking -- not nonlinearity itself -- as the active ingredient. Activation and normalization play distinct roles: activation concentrates continuity in the leading singular direction, while normalization distributes it across multiple directions. In transformers, continuity is projection-specific: Q, K, Gate, and Up (which read from the residual stream) develop input-space ($\mathbf{v}_1$) continuity; O and Down (which write to it) develop output-space ($\mathbf{u}_1$) continuity; V alone, lacking an adjacent nonlinearity, develops only low continuity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。