发现大模型能力涌现的两种反直觉增长模式。
U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Models
- 按题目难易分组，发现难题呈U型增长，易题先降后升
- 整体性能停滞因两种模式相互抵消，易题转为正向增长时突然跃升
- 提出切片夹心法预测能力突变阈值，代码开源
大型语言模型在某些下游任务中表现出能力涌现现象，即性能在规模增加初期保持平稳，随后在超过某个阈值后突然且不可预测地提升。本文通过按题目难度分组分析，发现难题呈现U型缩放规律，易题则先呈倒U型再趋于稳定提升。这两种模式初期相互抵消，导致整体性能停滞；当易题的缩放模式从倒置转为常规正向增长时，性能开始飙升，引发能力涌现。基于此，我们提出一种简单有效的预测管道Slice-and-Sandwich，用于预判能力涌现阈值及阈值后的模型表现。代码已公开于https://github.com/tony10101105/ExpEmergence。
原文摘要 · Abstract (English)
Large language models (LLMs) have been shown to exhibit emergent abilities in some downstream tasks, where model performance stagnates at first and then improves sharply and unpredictably with scale beyond a threshold. In this work, we investigate the phenomenon by grouping questions based on difficulty level and provide a possible explanation for emergent abilities. Specifically, we observe U-shaped scaling for hard questions and inverted-U scaling followed by steady improvement for easy questions. The two scaling patterns initially offset each other, causing stagnant overall performance. The performance starts to soar when the scaling pattern of easy questions reverts from inverse to standard scaling, leading to emergent abilities. Based on this finding, we propose a simple yet effective pipeline, called Slice-and-Sandwich, to predict the emergence threshold and model performance beyond the threshold. Our code is publicly available at https://github.com/tony10101105/ExpEmergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。