探索解码器模型在多语言翻译中的缩放规律，发现其性能可预测但有局限。
Scaling Laws of Decoder-Only Models on the Multilingual Machine Translation Task
- 用70M到7B参数的解码器模型在多语言数据上训练并测试
- 小模型表现符合大语言模型缩放规律，大模型和新数据分布则不适用
- 增加深度或宽度都能提升性能，但效率影响不同，适合模型优化研究者
近期研究表明，解码器仅模型在诸多自然语言处理任务中展现出卓越能力，包括机器翻译。然而，机器翻译领域长期由基于Transformer架构的编码器-解码器模型主导。尽管编码器-解码器模型的缩放规律已有深入研究，解码器仅模型仍较少被关注。本文在句子级、多语言、多领域的翻译任务上，训练了一系列从70M到7B参数的六种解码器仅模型。通过一系列实验表明，解码器仅模型的损失可使用与大型语言模型类似的缩放规律进行估计，但该规律在过于庞大的模型或不同数据分布下难以泛化。同时，本文研究了不同的缩放方法，发现增加模型深度与宽度能带来相似的测试损失改善，但对模型效率的影响各异。
原文摘要 · Abstract (English)
Recent studies have showcased remarkable capabilities of decoder-only models in many NLP tasks, including translation. Yet, the machine translation field has been largely dominated by encoder-decoder models based on the Transformer architecture. As a consequence, scaling laws of encoder-decoder models for neural machine translation have already been well studied, but decoder-only models have received less attention. This work explores the scaling laws of decoder-only models on the multilingual and multidomain translation task. We trained a collection of six decoder-only models, ranging from 70M to 7B parameters, on a sentence-level, multilingual and multidomain dataset. We conducted a series of experiments showing that the loss of decoder-only models can be estimated using a scaling law similar to the one discovered for large language models, but we also show that this scaling law has difficulties to generalize to too large models or to a different data distribution. We also study different scaling methods and show that scaling the depth and the width of a model lead to similar test loss improvements, but with different impact on the model's efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。