通过架构调控提升大模型推理效率，实现更高精度与速度。
Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
- 引入带架构信息的条件缩放定律，优化模型设计。
- 在相同训练预算下，精度提升2.1%，推理吞吐量提高42%。
- 适合追求高效推理的大模型研发与部署人员。
增大参数量和训练数据规模已被证明能有效提升大语言模型性能。然而，随着模型日益强大且广泛应用，推理成本成为关键挑战。本文研究隐藏层大小、MLP与注意力参数分配比例（mlp-to-attention ratio）以及分组查询注意力（GQA）等关键架构因素对推理效率和准确率的影响。我们提出一种融合架构信息的条件缩放定律，扩展了Chinchilla框架，并构建搜索框架以发现兼具推理高效与高精度的模型架构。通过训练超过200个模型（参数量80M至3B，训练样本8B至100B token），我们验证了该条件缩放定律的预测能力。结果表明，优化后的架构在同等训练预算下，相比LLaMA-3.2，精度最高提升2.1%，推理吞吐量提升42%。
原文摘要 · Abstract (English)
Scaling the number of parameters and the size of training data has proven to be an effective strategy for improving large language model (LLM) performance. Yet, as these models grow increasingly powerful and widely deployed, the cost of inference has become a pressing concern. Despite its importance, the trade-off between model accuracy and inference efficiency remains underexplored. In this work, we examine how key architectural factors, hidden size, the allocation of parameters between MLP and attention (mlp-to-attention ratio), and grouped-query attention (GQA), influence both inference cost and accuracy. We introduce a conditional scaling law that augments the Chinchilla framework with architectural information, along with a search framework for identifying architectures that are simultaneously inference-efficient and accurate. To validate our approach, we train more than 200 models spanning 80M to 3B parameters and 8B to 100B training tokens, and fit the proposed conditional scaling law. Our results show that the conditional scaling law reliably predicts optimal architectural choices and that the resulting models outperform existing open-source baselines. Under the same training budget, optimized architectures achieve up to 2.1% higher accuracy and 42% greater inference throughput compared to LLaMA-3.2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。