让Transformer模型可稳定扩展，无需重训就能高效扩容。
Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling
- 用非线性映射替代传统线性投影，突破维度限制。
- 440M参数扩展时，训练计算量比同类方法少41.5%。
- 支持零初始化新增模块，知识保留更稳定，适合持续迭代场景。
Transformer模型的扩展通常需从头训练，因标准架构难以在不丢弃已有表示的情况下拓展。我们发现注意力机制中线性投影是主要瓶颈——它将特征提取严格限制在固定维度子空间，限制了表达能力与增量扩展潜力。为此，提出Nexusformer，用Nexus-Rank层替代传统的线性Q/K/V投影，该层通过双激活在逐级升维空间中实现三阶段非线性映射。此设计打破线性约束，实现无损结构化增长：可通过零初始化模块沿两个方向注入新容量，同时保留预训练知识。在语言建模与推理基准上的实验表明，当从240M扩展至440M参数时，Nexusformer在达到Tokenformer同等困惑度的同时，训练计算量减少达41.5%。此外，对增长动力学的分析显示，零初始化诱导出稳定的收敛轨迹，由此推导出一个几何缩放律，能准确预测不同扩展尺度下的性能表现。
原文摘要 · Abstract (English)
Scaling Transformers typically necessitates training larger models from scratch, as standard architectures struggle to expand without discarding learned representations. We identify the primary bottleneck in the attention mechanism's linear projections, which strictly confine feature extraction to fixed-dimensional subspaces, limiting both expressivity and incremental capacity. To address this, we introduce Nexusformer, which replaces linear $Q/K/V$ projections with a Nexus-Rank layer, a three-stage nonlinear mapping driven by dual activations in progressively higher dimensional spaces. This design overcomes the linearity constraint and enables lossless structured growth: new capacity can be injected along two axes via zero-initialized blocks that preserve pretrained knowledge. Experiments on language modeling and reasoning benchmarks demonstrate that Nexusformer matches Tokenformer's perplexity using up to 41.5\% less training compute during progressive scaling (240M to 440M). Furthermore, our analysis of growth dynamics reveals that zero initialization induces a stable convergence trajectory, allowing us to derive a geometric scaling law that accurately predicts performance across expansion scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。