数据可预测性决定Transformer权重增长幅度,提前可算出训练结果。
Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training

- 用前训练的词频熵预估权重尺度增长,建立无需训练的预测公式。
- 实测23组不同学习率下,预测误差仅5.7%,相关系数达0.941。
- 适用于多种模型结构,揭示数据冗余是影响权重演化的重要因素。
Transformer训练后权重大小可用双参数威布尔分布描述,其形状参数 $k \approx 1.2$ 在层间与模型间稳定,尺度参数 $λ$ 反映训练过程中的主要变化。什么语料特性决定了 $λ$ 的增长?我们发现,利用前训练计算的二元组条件熵 $D = H(\text{next} \mid \text{prev})$,可建立一个与学习率相关的规律:$λ^2 - λ_0^2 = C_0(η) + C_1(η)(H_r - D)^{0.59}$,其中 $H_r$ 是同预算随机打乱基线。该凸指数源于独立测量的数据饱和关系,并非直接拟合生长曲线。去除每学习率对应的两个系数后,23次实验(学习率跨度一个数量级)均沿 $(H_r - D)^{0.59}$ 以单位斜率收敛($R^2 = 0.941$;按学习率单独拟合时 $R^2 \approx 0.82$)。因 $D$ 可在训练前计算,该规律具备前瞻性预测能力:端到端自验证对同家族未见数据的权重增长预测误差仅为5.7%。该规律在模型与层级别均成立,且跨两种架构保持函数形式,仅系数改变。同时揭示边界:跨语料预测会高估代码数据,暗示冗余是更广义 $Φ(D,R,A,H)$ 数据-权重框架的另一维度。
原文摘要 · Abstract (English)
A trained transformer's weight magnitudes can be summarized by a two-parameter Weibull distribution whose shape $k \approx 1.2$ is stable across layers and models, so the scale $λ$ carries most training-induced movement. What corpus property sets how much $λ$ grows? Using the bigram conditional entropy $D = H(\text{next} \mid \text{prev})$, a training-free statistic computed before training, we find across controlled corruption families a learning-rate-conditioned law, $λ^2 - λ_0^2 = C_0(η) + C_1(η)(H_r - D)^{0.59}$, where $H_r$ is a matched-budget shuffle baseline. The convex exponent is inherited from an independently measured data-side saturation relation rather than fitted directly to the growth curve. After removing the two per-$η$ coefficients, 23 runs spanning an order of magnitude in learning rate collapse onto $(H_r - D)^{0.59}$ with unit slope ($R^2 = 0.941$; direct per-$η$ fits are weaker, $R^2 \approx 0.82$). Because $D$ is computed before training, the law is a forward predictor: an end-to-end self-validation recovers held-out within-family weight growth with 5.7% relative error. The readout holds at model and per-layer resolutions and across two tested architectures, with the functional form preserved and only the coefficients changing. It also marks its boundary: cross-corpus prediction over-predicts code, implicating redundancy as a second axis of a broader $Φ(D,R,A,H)$ data-to-weight framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。