发现大模型实际用的层数远低于理论深度，且不随任务变难而增加。
What Affects the Effective Depth of Large Language Models?
- 通过分析不同规模模型，发现有效层数随模型增大但利用率不变。
- 长上下文推理提升来自更长输入而非更深计算，层数未增加。
- 模型不会因任务变难自动启用更多层，存在深层利用不足问题。
大规模语言模型（LLM）在扩展时强调增加深度，但性能增益随层数增加而减弱。已有研究提出“有效深度”概念，认为深层模型无法充分利用所有层进行有意义计算。本文系统研究了有效深度在模型规模、训练类型和任务难度下的变化。基于Qwen-2.5系列（1.5B–32B）模型分析发现，尽管有效层数随模型规模增长，但有效深度比例保持稳定。对比基础模型与长思维链（long-CoT）模型显示，有效深度未提升，表明推理能力增强主要源于更长上下文而非每标记的深层计算。跨不同难度任务评估表明，模型不会动态使用更多层应对复杂任务。结果表明当前LLM在各类场景下均未充分使用可用深度，提示未来可在提升层利用率、模型剪枝和早期退出方面展开研究。代码已开源：https://github.com/AheadOFpotato/what_affects_effective_depth。
原文摘要 · Abstract (English)
The scaling of large language models (LLMs) emphasizes increasing depth, yet performance gains diminish with added layers. Prior work introduces the concept of "effective depth", arguing that deeper models fail to fully utilize their layers for meaningful computation. Building on this, we systematically study how effective depth varies with model scale, training type, and task difficulty. First, we analyze the model behavior of Qwen-2.5 family (1.5B-32B) and find that while the number of effective layers grows with model size, the effective depth ratio remains stable. Besides, comparisons between base and corresponding long-CoT models show no increase in effective depth, suggesting that improved reasoning stems from longer context rather than deeper per-token computation. Furthermore, evaluations across tasks of varying difficulty indicate that models do not dynamically use more layers for harder problems. Our results suggest that current LLMs underuse available depth across scales, training paradigms and tasks of varying difficulties, pointing out research opportunities on increasing the layer utilization rate of LLMs, model pruning, and early exiting. Our code is released at https://github.com/AheadOFpotato/what_affects_effective_depth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。