提出兼顾推理效率的模型缩放法则，显著提升大模型推理速度。
Scaling Inference-Efficient Language Models
- 基于修正的缩放定律，联合优化参数量、训练数据与模型结构。
- 在相同精度下，推理延迟降低1.8倍，优于现有开源模型。
- 发现宽而浅的结构更高效，适合资源受限场景部署。
缩放定律是预测大语言模型性能的强大工具，但现有方法未能考虑推理成本。本文首先发现，相同规模的模型在推理延迟上可相差高达3.5倍，且相似训练损失的模型在下游任务中仍存在表现差异。为此，我们改进了Chinchilla缩放定律，使其同时优化模型参数量、训练数据量与架构设计，并提出一种基于新缩放定律的训练方法，以构建推理高效的模型。我们进行了广泛的实证研究，涵盖80M至1B参数、1.6B至30B训练样本及多种模型结构，共训练63个模型。基于提出的推理高效缩放定律与模型选择方法，我们发布了Morph-1B模型，在保持下游任务准确率的同时，推理延迟降低1.8倍，突破了准确率-延迟权衡的帕累托前沿。实验还表明，更宽更浅的模型可在不牺牲准确率的前提下实现效率提升。
原文摘要 · Abstract (English)
Scaling laws are powerful tools to predict the performance of large language models. However, current scaling laws fall short of accounting for inference costs. In this work, we first show that model architecture affects inference latency, where models of the same size can have up to 3.5x difference in latency. To tackle this challenge, we modify the Chinchilla scaling laws to co-optimize the model parameter count, the number of training tokens, and the model architecture. Due to the reason that models of similar training loss exhibit gaps in downstream evaluation, we also propose a novel method to train inference-efficient models based on the revised scaling laws. We perform extensive empirical studies to fit and evaluate our inference-aware scaling laws. We vary model parameters from 80M to 1B, training tokens from 1.6B to 30B, and model shapes, training 63 models. Guided by our inference-efficient scaling law and model selection method, we release the Morph-1B model, which improves inference latency by 1.8x while maintaining accuracy on downstream tasks compared to open-source models, pushing the Pareto frontier of accuracy-latency tradeoff. Notably, our experiments reveal that wider and shallower models can yield efficiency gains while preserving accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。