通过重塑权重分布提升LoRA训练效果，无需修改模型结构。
Exploring and Reshaping the Weight Distribution in LLM
- 发现不同层间权重余弦距离呈幂律分布，揭示内在规律。
- 基于幂律特性设计生成器，模拟符合分布的初始权重。
- 仅调整LoRA初始化权重，即实现训练性能提升。
大型语言模型的性能受架构、模型规模、解码方法等因素影响，不同层的权重分布存在差异。本文研究了权重分布与各层类型间的关联，并探讨其对LoRA训练效果的影响。首先，发现模型中不同层权重间的余弦距离服从幂律分布；从自注意力与MLP层提取Query投影、下投影等权重矩阵，通过奇异值分解计算其奇异值，并按投影类型组织为矩阵。分析表明，这些矩阵间的余弦距离具有显著的幂律分布特征。其次，基于层间距离分析结果，提出一种定性描述不同模型分布特性的方法。进一步，结合高斯过程与帕累托分布函数，设计数据生成器以模拟符合特定分布特征的数据。最后，利用该分布特性与生成方法重构LoRA初始化权重。实验表明，不改变模型结构或训练流程，该方法可有效提升LoRA训练性能。
原文摘要 · Abstract (English)
The performance of Large Language Models is influenced by their characteristics such as architecture, model sizes, decoding methods and so on. Due to differences in structure or function, the weights in different layers of large models have varying distributions. This paper explores the correlations between different types of layers in terms of weights distribution and studies the potential impact of these correlations on LoRA training effectiveness. Firstly, the study reveals that in the model the cosine distances between weights of different layers manifest power-law distribution. We extract Query-projection, down-projection and other weight matrices from the self-attention layers and MLP layers, calculate the singular values of the matrices using singular value decomposition, and organize a certain number of singular values into matrices according to projection's type. By analyzing the probability distribution of the cosine distances between these matrices, it is found that the cosine distances values between them have distinct power-law distribution characteristics. Secondly, based on the results of distance calculations and analysis across different layers of model, a qualitative method is proposed to describe the distribution characteristics of different models. Next, to construct weights that align with the distribution characteristics, a data generator is designed using a combination of Gaussian process and Pareto distribution functions. The generator is used to simulate the generation of data that aligns with specific distribution characteristics. Finally, based on the aforementioned distribution characteristics and data generation method, the weights in LoRA initialization are reshaped for training. Experimental results indicate that, without altering the model structure or training process, this method achieves a certain improvement in the performance of LoRA training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。