提出双流结构SiameseNorm，解决Transformer中预归一化与后归一化的根本矛盾。
SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm
- 设计双流架构，共享残差块实现预/后归一化信号协同优化。
- 在400M至15B模型上提升性能，训练稳定性显著增强。
- 适用于语言、视觉及扩散模型，兼容现有预归一化训练流程。
Transformer架构中预归一化（Pre-Norm）与后归一化（Post-Norm）之间的长期矛盾仍是一个未解难题，反映了训练稳定性与表征能力间的根本权衡。此前尝试融合两者优势虽有进展，但跨训练场景鲁棒性有限，应用受限。本文重新审视该问题，发现单流架构难以同时实现预归一化的稳定梯度传播与后归一化的主残差路径归一化。为此，提出简单高效的双流结构SiameseNorm，通过共享残差块连接预归一化与后归一化路径，使每个残差块可接收双路径优化信号，开销极低。在400M与1.3B稠密语言模型、15B MoE模型、视觉Transformer及扩散Transformer上广泛实验表明，SiameseNorm在多种架构与模态下均持续提升性能，同时保持强训练稳定性。代码已开源：https://github.com/Qwen-Applications/SiameseNorm。
原文摘要 · Abstract (English)
The long-standing tension between Pre- and Post-Norm remains an open problem in Transformer architecture, reflecting a fundamental trade-off between training stability and representational capacity. Prior attempts to combine their strengths have made progress, but often show limited robustness across training settings, restricting their broader applicability. We revisit this dilemma, showing that single-stream architectures struggle to reconcile Pre-Norm's stable identity-gradient propagation with Post-Norm's normalization of the main residual path. To address this structural tension, we propose SiameseNorm, a simple yet effective two-stream architecture that remains compatible with Pre-Norm training recipes. SiameseNorm couples Pre-Norm-like and Post-Norm-like streams through shared residual blocks, allowing each residual block to receive optimization signals from both pathways with negligible overhead. Extensive experiments on 400M and 1.3B dense language models, 15B MoE models, Vision Transformers, and Diffusion Transformers show that SiameseNorm consistently improves performance while maintaining strong training stability across architectures and modalities. Code is available at https://github.com/Qwen-Applications/SiameseNorm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。