提出新指标DOCS，量化大模型权重相似性，揭示层间功能分工。
DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models
- 用余弦相似度分布（DOCS）量化大模型各层权重相似性。
- 发现相邻层权重高度相似并形成簇，暗示分层功能专化。
- 理论证明对正交初始化有效，适合研究模型结构与可解释性。
我们提出一种新指标——余弦相似度分布（DOCS），用于定量评估大语言模型（LLMs）中权重矩阵间的相似性，旨在促进对其复杂架构的分析。基于DOCS的分析揭示了最新开源大模型中的有趣模式：相邻层常表现出高权重相似性，并倾向于形成聚类，暗示深度方向上的功能专化。此外，我们证明了DOCS在正交矩阵相似性量化上具有理论有效性，而正交初始化在大模型中极为普遍。该研究深化了对大模型架构与行为的理解，为构建更高效、可解释的模型提供了潜在工具。
原文摘要 · Abstract (English)
We introduce a novel index, the Distribution of Cosine Similarity (DOCS), for quantitatively assessing the similarity between weight matrices in Large Language Models (LLMs), aiming to facilitate the analysis of their complex architectures. Leveraging DOCS, our analysis uncovers intriguing patterns in the latest open-source LLMs: adjacent layers frequently exhibit high weight similarity and tend to form clusters, suggesting depth-wise functional specialization. Additionally, we prove that DOCS is theoretically effective in quantifying similarity for orthogonal matrices, a crucial aspect given the prevalence of orthogonal initializations in LLMs. This research contributes to a deeper understanding of LLM architecture and behavior, offering tools with potential implications for developing more efficient and interpretable models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。