发现大模型权重始终服从高斯分布，且可由噪声生成。
Unveiling the Mystery of Weight in Large Foundation Models: Gaussian Distribution Never Fades
- 从噪声中生成变换权重，扩大权重容差范围。
- 权重标准差随层数加深而增大，利于下游任务适应。
- 最优权重应零均值、对称、稀疏，呈截断高斯分布。
本文首次系统探索大基础模型（LFM）权重的内在机制，旨在简化人工智能研究。通过对主流LFM的广泛观察与分析，发现无论初始化方式如何，其权重主要呈现高斯分布，偶有尖锐的倒T形或线性模式。我们进一步发现权重具有高斯噪声的独立同分布特性，并揭示其直接关联：变换权重可由高斯噪声生成，主要作用是提升预训练权重的标准差，且该标准差随网络深度增加而增大。这意味着变换权重扩大了对最优权重的容许偏差，有助于模型适应下游任务。基于上述结论，我们深入探讨了最优权重的本质，最终认为其应具备零均值、对称性与稀疏性，稀疏值为截断高斯分布，仅含少量异常值。在模型适配与编辑的实验中验证了这些洞见的有效性。期望这些发现能为未来大模型研究提供基础理解。
原文摘要 · Abstract (English)
This paper presents a pioneering exploration of the mechanisms underlying large foundation models' (LFMs) weights, aiming to simplify AI research. Through extensive observation and analysis on prevailing LFMs, we find that regardless of initialization strategies, their weights predominantly follow a Gaussian distribution, with occasional sharp, inverted T-shaped, or linear patterns. We further discover that the weights share the i.i.d. properties of Gaussian noise, and explore their direct relationship. We find that transformation weights can be derived from Gaussian noise, and they primarily serve to increase the standard deviation of pre-trained weights, with their standard deviation growing with layer depth. In other words, transformation weights broaden the acceptable deviation from the optimal weights, facilitating adaptation to downstream tasks. Building upon the above conclusions, we thoroughly discussed the nature of optimal weights, ultimately concluding that they should exhibit zero-mean, symmetry, and sparsity, with the sparse values being a truncated Gaussian distribution and a few outliers. Our experiments in LFM adaptation and editing demonstrate the effectiveness of these insights. We hope these findings can provide a foundational understanding to pave the way for future advancements in the LFM community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。