解决RMSNorm模型坐标对齐的符号与排列问题，提升跨检查点迁移精度。
Signed-Permutation Coordinate Transport for RMSNorm Transformers
- 提出符号-排列匹配算法，修复RMSNorm模型的对称性缺陷
- 坐标保真传输使1500步时坐标恢复率达91.1%，远超传统方法
- 适用于模型编辑、可解释性分析与状态续训等需要精确坐标对齐场景
现代大模型工作流需在检查点间传递坐标索引对象，如引导向量、稀疏自编码器、顶级神经元集合等。这要求先固定残差流的规范对称性。我们发现该对称性依赖于架构：LayerNorm的规范对称性为置换群 $S_d$（允许全局符号翻转），而通用通道增益的RMSNorm则具有符号-排列群 $B_d = S_d times \{\pm 1\}^d$。因此仅使用置换对齐对RMSNorm模型不完整。本文引入符号边际匈牙利匹配，并证明：在坐标无关时，原始带符号相关匹配存在结构性准确上限，其值等于真实规范中正号比例；该上限可通过符号边际化消除。随后，以坐标保真传输为核心目标，沿同一基础微调轨迹组合保存检查点的 $B_d$ 规范，1500步时坐标恢复率达91.1%（端点匹配仅60.3%），且非单纯路径路由所致。该恢复的规范使原本因置换对齐失效的工具得以复用：TinyLlama SAE重建的NMSE从1.08降至0.004；Qwen情感引导保留率从17.2%升至95.8%；拒绝引导在 $S_d$ 下符号反转。状态训练亦受此影响：采用符号传输的AdamW状态能复现原轨迹，而置换对齐则偏离。最后，规范扫描审计显示，索引级可解释性结论仅在显式规范下可重复。
原文摘要 · Abstract (English)
Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top-$k$ neuron sets, attribution lists, and merge alignments. This is only well posed after fixing the model's residual-stream gauge, which we show is architecture-dependent: LayerNorm residual charts have permutation gauge $S_d$ (up to a global sign flip), while RMSNorm charts with generic per-channel gain have signed-permutation gauge $B_d = S_d \ltimes \{\pm 1\}^d$. Permutation-only alignment is therefore symmetry-incomplete for RMSNorm models. We introduce sign-marginalized Hungarian matching and prove a sharp failure mode: with decorrelated coordinates, raw signed-correlation matching has a structural permutation-accuracy ceiling at the positive-sign fraction of the true gauge, which sign-marginalization removes. We then make coordinate-preserving transport, not function-level merging, the primary object: composing saved-checkpoint local $B_d$ gauges along same-base fine-tuning trajectories recovers 91.1% of cross-run coordinates at 1500 steps versus 60.3% for endpoint matching, and the gain is not explained by merely routing through the base. The recovered gauge transfers tools that permutation-only alignment breaks: TinyLlama SAE reconstruction has NMSE 0.004 under $B_d$ versus 1.08 under $S_d$; Qwen sentiment steering preserves 95.8% of its effect versus 17.2%; refusal steering reverses sign under $S_d$; coordinate-preserving merges behave the same way. The same covariance governs stateful training: signed transport of AdamW state preserves the resumed trajectory, while permutation-only state follows a different one from a functionally identical checkpoint. Finally, gauge-sweep audits show index-level interpretability claims are reproducible only relative to an explicit gauge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。