探索复数循环语言模型的演化,揭示SWAVE设计缺陷并提出稳定训练新方法。
Why SWAVE May Not Be All You Need:A Concept-Evolution Retrospective on Complex-Valued Recurrent Language Models
- 用相位关联记忆架构替代原结构,解决虚部坍塌问题
- 实现20万步稳定训练,最低困惑度达22.0
- 提炼6条复数递归训练工程原则,适合长期序列建模研究者
SWave是一个参数量169.26M、维度D=384、层数L=16、上下文长度T=2048的复数循环语言模型,基于FineWeb-Edu数据集在2xH100 NVL上训练。其核心设计依赖三个前提:复数表示可更丰富编码信息;Cayley参数化酉转移保证状态不发散;旋转隐藏状态保持长程信号完整。研究发现,共振头存在虚部坍塌的全局极小值(称作cos-坍塌),被无绑定头取代。复杂归一化和波传播扫描贯穿三阶段,保留至最终架构。ProtectGatedScan转为结构先验,多尺度保留机制未显优势,复数门控单元被实数平方ReLU混合器替代。辅助目标在结构约束解决后无效。研究揭示了cos-坍塌的数学本质,提出对数空间反向传播以提升数值稳定性,提炼六条可迁移工程原则,并建立计划到代码的可追溯性方法,能识别传统测试遗漏的结构偏差。
原文摘要 · Abstract (English)
SWave is a complex-valued recurrent language model (169.26M parameters, D=384, L=16, T=2048) trained on FineWeb-Edu using 2xH100 NVL. It was designed around three founding premises: that representing language as complex waves rather than real-valued numbers enables richer information encoding; that a Cayley-parameterised unitary transition provides a mathematical guarantee against state decay or explosion; and that a hidden state which rotates rather than shrinks preserves signal integrity over arbitrarily long contexts. The core of SWave evolved substantially across three development phases. The Resonance Head was found to structurally admit imaginary-channel collapse as a global loss minimum (a failure mode we term cos-domination collapse) and was superseded by an untied head with independent real and imaginary embedding tables from the Phase-Associative Memory (PAM) architecture. This resolved the degenerate minimum and enabled stable 200,000-step training (best-step PPL 22.0 at step 89,861). ComplexNorm and the Wave Propagation Scan proved load-bearing throughout all three phases and were retained to the final architecture. ProtectGatedScan was reframed as a structural prior rather than a learned behaviour. The four multi-scale retention concepts showed no measurable improvement under controlled evaluation and were found non-load-bearing. The ComplexGatedUnit was superseded by a real-valued squared-ReLU channel mixer with fewer parameters. The auxiliary training objectives showed no benefit once structural constraints were resolved. The investigation yields a formal characterisation of cos-domination collapse, a parallel scan with a log-space backward pass for numerical stability, six transferable engineering principles for complex-valued recurrent training, and a plan-to-code traceability methodology for catching structural divergences that conventional test suites miss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。