探究大模型设计对下游性能的影响，发现非规模因素更关键
Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions
- 用92个开源模型做元分析，结合多维度设计特征预测性能
- 引入数据构成、架构选择等变量后，预测准确率提升3%-28%
- 揭示代码比例15%-25%是语言与代码任务的平衡点
语言模型能力提升常归因于模型规模或训练数据量，但某些小模型通过精心筛选数据或特殊架构设计，反而优于更大模型。为量化这些设计决策的影响，我们对92个开源预训练模型进行了元分析，涵盖从顶尖开放权重模型到非主流设计的各类模型。结果表明，加入模型规模和训练样本数以外的特征后，对下游性能的预测能力相对提升3%-28%。分析显示，数据构成中代码占比15%-25%是语言与代码任务间的平衡点；某些架构选择如旋转位置编码优于学习型嵌入。本研究为系统理解模型开发决策如何塑造最终能力奠定了基础。
原文摘要 · Abstract (English)
Improvements in language model capabilities are often attributed to increasing model size or training data, but in some cases smaller models trained on curated data or with different architectural decisions can outperform larger ones trained on more tokens. What accounts for this? To quantify the impact of these design choices, we meta-analyze 92 open-source pretrained models across a wide array of scales, including state-of-the-art open-weights models as well as less performant models and those with less conventional design decisions. We find that by incorporating features besides model size and number of training tokens, we can achieve a relative 3-28% increase in ability to predict downstream performance compared with using scale alone. Analysis of model design decisions reveal insights into data composition, such as the trade-off between language and code tasks at 15-25\% code, as well as the better performance of some architectural decisions such as choosing rotary over learned embeddings. Broadly, our framework lays a foundation for more systematic investigation of how model development choices shape final capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。