arXiv:2605.15572cs.CL2026-05

测量现代开源大模型激活值最大范围,揭示其与架构、训练阶段密切相关。

Measuring Maximum Activations in Open Large Language Models

论文配图:Measuring Maximum Activations in Open Large Language Models
图 1 · 摘自论文原文
  • 统一流程量化27个模型的逐层激活峰值,覆盖8个家族
  • 峰值跨度近4个数量级,最高达70万,MoE模型峰值低14-23倍
  • 残差流常含全局最大值,建议发布时报告激活范围

激活值动态范围是低比特量化、激活缩放和稳定LLM推理的一阶约束。以往研究聚焦于2024年前的LLaMA类模型,但对后LLaMA时代开源模型的激活特性缺乏重新评估。我们提出面向部署的问题:现代开源大模型中激活值最大能到多大?其大小在不同模型家族、代际和训练阶段如何变化?在统一流程(5,000样本多领域语料,家族特定分词,统一钩子覆盖嵌入、隐藏状态、注意力、MLP/MoE、SwiGLU门控和最终归一化)下,我们测量了27个检查点(来自8个开源家族,涵盖密集型、MoE、视觉语言、中间训练、指令微调等变体)的全局与逐层最大值。发现:(i) 全局最大值在相近参数量下跨越近四个数量级,Qwen3.5和MoE检查点为10²~10³量级,Gemma3-27B-it达约7×10⁵;(ii) 不同家族与代际间无简单单调关系;(iii) MoE检查点峰值比同规模密集型低14.0–23.4倍,且残差流在22/24个检查点中承载全局最大值。轻量级INT-8验证显示,测得峰值与低比特重构误差通过激活缩放选择共变。结论:最大激活值是受家族、架构和训练阶段影响的模型属性,而非单纯规模产物,应在任何开放权重发布前测量并报告,以支持低比特部署。代码已公开于https://github.com/clx1415926/Max_act_llm。

原文摘要 · Abstract (English)

The dynamic range of activations is a first-order constraint for low-bit quantization, activation scaling, and stable LLM inference. Prior work characterized outlier features and massive activations on pre-2024 LLaMA-style models, and the downstream activation-quantization stack inherits that picture without revisiting it for the post-LLaMA open-model boom. We ask the deployment-oriented question: how large can activations get in modern open LLMs, and how does this magnitude vary across families, generations, and training stages? Under a unified pipeline (5,000-sample multi-domain corpus, family-specific tokenization, identical hooks across embeddings, hidden states, attention, MLP/MoE, SwiGLU gates, and final norm), we measure global and layerwise maxima on 27 checkpoints from 8 open families spanning dense, MoE, vision-language, intermediate-training, and instruction-tuned variants. We find that (i) global maxima span over nearly four orders of magnitude at comparable parameter counts, with Qwen3.5 and MoE checkpoints in the 10^2 to 10^3 range and Gemma3-27B-it reaching ~7 x 10^5; (ii) cross-family and cross-generation comparisons break simple monotonic scaling; and (iii) MoE checkpoints exhibit 14.0-23.4x lower peaks than matched-scale dense counterparts, while the residual stream carries the global maximum in 22/24 checkpoints. A lightweight INT-8 sanity check shows that measured maxima co-vary with low-bit reconstruction error via activation-scale selection. We conclude that maximum activation magnitude is a model property tied to family, architecture, and training stage - not a simple byproduct of size - and should be measured and reported alongside any open-weight release before low-bit deployment. The code is publicly available at https://github.com/clx1415926/Max_act_llm.

大模型量化激活值性能分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。