系统分析大模型结构对性能的影响，揭示可优化规律。
From Parameters to Performance: A Data-Driven Study on LLM Structure and Development
- 构建包含多种开源模型结构的性能数据集。
- 发现层数、头数等配置与多任务表现显著相关。
- 适合模型设计者和架构研究者参考。
大型语言模型（LLMs）在多个领域取得显著成功，推动了技术进步。尽管模型规模和能力快速提升，但关于结构配置如何影响性能的系统性、数据驱动研究仍较少。为此，我们构建了一个大规模数据集，涵盖多样化的开源LLM结构及其在多个基准上的表现。基于该数据集，我们开展系统性的数据挖掘分析，验证并量化结构配置与性能之间的关系。研究回顾了LLM的历史发展，探索未来趋势；分析不同结构选择对基准表现的影响，并通过机制可解释性技术进一步验证结果。本工作为大模型优化提供数据驱动洞见，旨在指导未来模型的针对性研发与应用。数据集将公开于 https://huggingface.co/datasets/DX0369/LLM-Structure-Performance-Dataset。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved remarkable success across various domains, driving significant technological advancements and innovations. Despite the rapid growth in model scale and capability, systematic, data-driven research on how structural configurations affect performance remains scarce. To address this gap, we present a large-scale dataset encompassing diverse open-source LLM structures and their performance across multiple benchmarks. Leveraging this dataset, we conduct a systematic, data mining-driven analysis to validate and quantify the relationship between structural configurations and performance. Our study begins with a review of the historical development of LLMs and an exploration of potential future trends. We then analyze how various structural choices impact performance across benchmarks and further corroborate our findings using mechanistic interpretability techniques. By providing data-driven insights into LLM optimization, our work aims to guide the targeted development and application of future models. We will release our dataset at https://huggingface.co/datasets/DX0369/LLM-Structure-Performance-Dataset
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。