发现大模型深度增加时损失反比下降，因层间功能相似而非逐层累积。
Inverse Depth Scaling From Most Layers Being Similar
- 通过分析大模型与简化残差网络，发现深度与损失呈反比关系。
- 深度每翻倍，损失降低约20%，但效率低且依赖层间相似性。
- 适合关注模型架构设计与训练效率的科研人员阅读。
神经网络缩放定律描述了大型语言模型（LLMs）中损失与模型规模的关系，但深度与宽度对性能的影响可能不同，需更细致研究。本文通过分析大语言模型和简化残差网络，发现大模型中损失与深度呈反比关系。这一现象可能源于功能相似的层通过集成平均减少误差，而非通过组合式学习或离散化平滑动态实现。该机制虽效率低下但具有鲁棒性，可能源于残差网络的结构偏见以及目标函数与平滑动态不兼容。研究结果表明，提升大模型效率需在架构上创新，以促进深度的组合式利用。
原文摘要 · Abstract (English)
Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies. Here, we quantify how depth affects loss via analysis of LLMs and toy residual networks. We find loss scales inversely proportional to depth in LLMs, probably due to functionally similar layers reducing error through ensemble averaging rather than compositional learning or discretizing smooth dynamics. This regime is inefficient yet robust and may arise from the architectural bias of residual networks and target functions incompatible with smooth dynamics. The findings suggest that improving LLM efficiency may require architectural innovations to encourage compositional use of depth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。