发现大激活涌现有可预测数学规律，设计时即可预控。
Hidden Dynamics of Massive Activations in Transformer Training
- 用指数调制对数函数拟合激活变化，五参数精准描述
- 模型规模与训练阶段决定激活峰值，规律稳定可复现
- 仅凭架构就能预测激活行为，助优化训练与模型设计
本文首次系统分析了Transformer训练中大规模激活的发展过程，以Pythia模型族为实验平台，并公开全部数据集以支持后续研究。通过对多个模型规模和训练检查点的系统分析，我们发现大规模激活的出现遵循高度可预测的数学规律，可用一个五参数的指数调制对数函数准确建模。此外，我们构建了一个机器学习框架，仅从架构规格出发即可预测这些数学参数，对稳态行为预测精度高，对涌现时机和幅度预测中等。这些发现表明，大规模激活的出现受模型设计支配，可在训练前预见甚至控制，对模型稳定性、训练周期长度、可解释性与优化具有重要意义。代码已开源：https://github.com/Aimpoint-Digital/massive-activations-fork
原文摘要 · Abstract (English)
We present the first comprehensive analysis of massive activation development throughout transformer training, using the Pythia model family as our testbed, and release our full dataset publicly to support further research. Through systematic analysis of various model sizes across multiple training checkpoints, we demonstrate that massive activation emergence follows highly predictable mathematical patterns that can be accurately modeled using an exponentially-modulated logarithmic function with five key parameters. Additionally, We develop a machine learning framework to predict these mathematical parameters from architectural specifications alone, achieving high accuracy for steady-state behavior and moderate accuracy for emergence timing and magnitude. These findings enable architects to predict and potentially control key aspects of massive activation emergence through design choices, with significant implications for model stability, training cycle length, interpretability, and optimization. Our findings demonstrate that the emergence of massive activations is governed by model design and can be anticipated, and potentially controlled, before training begins. Code is available at https://github.com/Aimpoint-Digital/massive-activations-fork
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。