发现大模型计算分两阶段：早期稀疏核心+后期密集精修
Tracing Computation Density in LLMs
- 用s-Trace方法定位最优子图，高效逼近完整模型输出
- 早期小规模子图可复现输出头部分布，后期节点逐步优化精度
- 计算量与模型不确定性强相关，稀疏子图捕捉基础统计特征
基于Transformer的大语言模型包含数十亿参数构成的深层宽广计算图，但并非所有输入都充分使用其全部能力。本文提出s-Trace方法，高效估算大小为s的子图以最佳逼近完整模型输出。结果显示，多种大模型的计算呈现两个显著阶段：由早期层节点组成的微型子图即可重构完整输出分布的头部；后续添加更多位于后期层的节点（尤其是注意力头），逐步精细逼近完整输出分布。此外，每输入所需计算量与模型不确定性正相关，而更稀疏的子图仅编码浅层统计特征，如一元频次。整体表明，有效计算具有稳定的模块化结构：稀疏的早期核心提供粗略预测，再经后期更密集计算逐步优化。
原文摘要 · Abstract (English)
Transformer-based large language models (LLMs) are comprised of billions of parameters arranged in deep and wide computational graphs, but it is not clear that they exploit their full capacity for all inputs. We introduce the s-Trace method to efficiently estimate the subgraph of size s that best approximates a full model output. With this method, we find the computation in a variety of LLMs to be organized in two distinct phases. A small subgraph mostly composed of early-layer nodes can reconstruct the head of the full model output distribution. Adding further nodes, mostly located in later layers and increasingly consisting of attention heads, leads to incremental refinements in approximating the full output distribution. We find moreover that the amount of necessary computation per input correlates with model uncertainty, and that sparser subgraphs encode shallow statistics, such as unigram frequency. Overall, our results suggest a consistent modular organization in effective LLM computation, with a sparse early-layer core providing a rough prediction that is further refined through denser computations in later layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。