深度不是越深越好，宽模型比深模型更高效。
The Depth Delusion: Why Transformers Should Be Wider, Not Deeper
- 提出架构相关缩放定律，揭示宽度增长快于深度。
- 发现临界深度：层数超限反而增加损失，称'深度幻觉'。
- 适合大模型设计者参考，尤其关注参数效率时。
神经网络缩放定律描述了语言模型损失随参数量和数据量下降的规律，但将架构视为可互换——百亿参数可能来自浅而宽（10层，8192隐藏维度）或深而窄（80层，2048隐藏维度）的结构。我们提出架构条件缩放定律，分解其依赖关系，发现最优深度与容量的幂次为0.12，最优宽度为0.34，即宽度应以2.8倍速度超过深度。我们发现关键深度现象：当层数超过临界值（约等于宽度的0.44次方），增加层数反而提升损失，称为‘深度幻觉’。我们在30个跨1700万至70亿参数的Transformer架构上验证，每个在高算力样本上训练，达到决定系数R²=0.922。核心发现：在70亿规模下，64层模型（63.8亿参数）比32层模型（68.6亿参数）低0.12纳特，尽管更深。证明最优深度-宽度权衡在生产级仍成立。
原文摘要 · Abstract (English)
Neural scaling laws describe how language model loss decreases with parameters and data, but treat architecture as interchangeable--a billion parameters could arise from a shallow-wide model (10 layers & 8,192 hidden dimension) or a deep-narrow one (80 layers & 2,048 hidden dimension). We propose architecture-conditioned scaling laws decomposing this dependence, finding that optimal depth scales as D* ~ C^0.12 while optimal width scales as W* ~ C^0.34, meaning width should grow 2.8x faster than depth. We discover a critical depth phenomenon: beyond D_crit ~ W^0.44 (sublinear in W), adding layers increases loss despite adding parameters--the Depth Delusion. Empirically, we validate these findings across 30 transformer architectures spanning 17M to 7B parameters, each trained on representative high-compute samples, achieving R^2 = 0.922. Our central finding: at 7B scale, a 64-layer model (6.38B params) underperforms a 32-layer model (6.86B params) by 0.12 nats, despite being significantly deeper. This demonstrates that optimal depth-width tradeoffs persist at the production scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。