用威布尔分布分析大模型权重，揭示训练动态与模块功能差异。
A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions
- 引入双参数威布尔分布诊断权重分布，可逐层逐步追踪训练过程。
- 传输类权重的形状参数k稳定在1.186~1.204之间，跨架构一致。
- 选择类权重偏离威布尔分布，其偏移程度反映注意力存储方式差异。
本文将极值理论中的威布尔分布（双参数）作为变压器模型中元素级权重幅度分布的诊断框架。初始化时，独立同分布高斯权重服从半正态分布，通过中80%概率图拟合得k ≈ 1.20（本研究采用的协议）。这一基准使k成为不受架构影响的训练动态度量标准；对每个检查点各层的权重矩阵独立拟合，实现组件级、层级和步骤级的精细诊断，突破传统聚合统计的局限。对涵盖7种架构家族（Pythia, OLMo-1/2, LLaMA-3, Mistral, Qwen2.5/3）共12个模型的分析发现：第一，前馈网络模块与注意力输出投影（传输类）的k值高度集中，终端中位数在[1.186, 1.204]之间（跨家族变异系数0.51%），覆盖SwiGLU/GeLU激活函数、Pre-LN/QK-Norm结构及70M-14B规模；第二，注意力输入投影（选择类）偏离威布尔分布，偏离程度由存储方式决定：分离存储Q/K（OLMo-1, OLMo-2）导致k ∈ [0.76, 0.99]（严重）；分组查询注意力（GQA）模型中k ∈ [1.10, 1.16]（轻微）；Pythia的合并式W_qkv处于过渡区，其k随训练预算T/tau单调变化；第三，λ在训练中显著增长，在Pythia家族中与√(η/λ_wd)呈强相关（皮尔逊r=0.94，三类传输类），方向与Fan等（2025）一致。两个参数承载独立信息：k标识功能类别，λ指示训练进度。代码与数据集已发布于https://github.com/tiexinding/NPM-Weibull-public。
原文摘要 · Abstract (English)
We apply the Weibull distribution -- a two-parameter family from extreme-value theory -- as a diagnostic framework for element-wise weight magnitude distributions in transformers. At initialization, i.i.d. Gaussian weights give |w| ~ HalfNormal, yielding k ~ 1.20 via middle-80% probability-plot fit (the protocol used throughout this work). This anchor makes k a principled, architecture-independent measuring stick for training dynamics; fitting each weight matrix independently at every layer at every checkpoint enables per-component, per-layer, and per-step diagnostics that aggregate statistics cannot resolve. Applying this framework to 12 model entries spanning 7 architectural families (Pythia, OLMo-1/2, LLaMA-3, Mistral, Qwen2.5/3) reveals three findings. First, FFN modules and the attention output projection W_o -- the Transmission Class -- fall in a narrow k band: median terminal k in [1.186, 1.204] across 12 entries (cross-family CV = 0.51%), shared across SwiGLU/GeLU activations, Pre-LN/QK-Norm placements, and 70M-14B sizes. Second, the attention input projections W_q, W_k -- the Selection Class -- depart from the Weibull family, with severity shaped by storage: separately-stored Q/K (OLMo-1, OLMo-2) yields k in [0.76, 0.99] (deep); GQA models yield k in [1.10, 1.16] (mild); Pythia's merged W_qkv occupies a transitional zone tracking training budget T/tau monotonically. Third, lambda grows substantially during training and scales with sqrt(eta/lambda_wd) within the Pythia family (Pearson r = 0.94, three Transmission kinds), directionally consistent with Fan et al. (2025). The two parameters carry independent information: k labels the functional class, lambda labels training progress. We release npm-weibull-py v0.4 (Python library) and DATABASE_v9_1 at https://github.com/tiexinding/NPM-Weibull-public .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。