发现大模型参数越专业，知识存储越高效。
The Rise of Parameter Specialization for Knowledge Storage in Large Language Models
- 分析20个开源大模型，发现高级模型的MLP参数更专注存储特定类型知识。
- 参数专业化使知识利用效率显著提升，实验验证其因果作用。
- 适合关注模型内部机制与知识存储优化的研究者。
随着一系列大型语言模型的涌现，研究者们致力于在有限参数规模下最大化模型性能。然而，从微观视角看，关于如何更有效地将知识存储于模型参数中（尤其是MLP层）的研究仍较为匮乏。本文分析了20个公开可用的开源大语言模型，探究其强性能与参数中知识存储方式之间的关系。结果表明，随着模型能力增强，其参数表现出更高的专业化特征：MLP中的参数更倾向于编码相似类型的知识。实验验证，这种知识分布的专门化有助于提升模型对存储知识的利用效率。进一步通过因果训练实验确认，该专门化分布对模型高效利用知识起关键作用。
原文摘要 · Abstract (English)
Over time, a growing wave of large language models from various series has been introduced to the community. Researchers are striving to maximize the performance of language models with constrained parameter sizes. However, from a microscopic perspective, there has been limited research on how to better store knowledge in model parameters, particularly within MLPs, to enable more effective utilization of this knowledge by the model. In this work, we analyze twenty publicly available open-source large language models to investigate the relationship between their strong performance and the way knowledge is stored in their corresponding MLP parameters. Our findings reveal that as language models become more advanced and demonstrate stronger knowledge capabilities, their parameters exhibit increased specialization. Specifically, parameters in the MLPs tend to be more focused on encoding similar types of knowledge. We experimentally validate that this specialized distribution of knowledge contributes to improving the efficiency of knowledge utilization in these models. Furthermore, by conducting causal training experiments, we confirm that this specialized knowledge distribution plays a critical role in improving the model's efficiency in leveraging stored knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。