对比解码器和编码器-解码器结构，发现前者在推理效率上更有优势。
Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model
- 用前缀语言模型训练编码器-解码器架构，提升其性能表现
- 在150M至80亿参数规模下，性能与单向解码器相当甚至更优
- 推理效率更高，适合需要低延迟的应用场景
近期大语言模型研究从编码器-解码器架构转向主流的单向解码器架构，但这一转变缺乏从扩展性角度的严谨对比，引发对编码器-解码器模型潜力被低估的担忧。为此，我们重新审视编码器-解码器大模型（RedLLM），引入解码器模型（DecLLM）的最新训练方法。通过在约1.6万亿词的RedPajama V1数据集上预训练，并使用FLAN进行指令微调，我们在150M至80亿参数规模范围内进行了全面对比。结果表明，尽管解码器模型在预训练阶段更具计算效率，但红链模型（RedLLM）展现出媲美且可扩展的性能与上下文长度外推能力。指令微调后，其在多个下游任务中表现相当甚至更优，同时具备显著更高的推理效率。我们希望这些发现能推动对编码器-解码器架构的再评估，释放其构建高效强大模型的潜力。
原文摘要 · Abstract (English)
Recent large language model (LLM) research has undergone an architectural shift from encoder-decoder modeling to nowadays the dominant decoder-only modeling. This rapid transition, however, comes without a rigorous comparative analysis especially \textit{from the scaling perspective}, raising concerns that the potential of encoder-decoder models may have been overlooked. To fill this gap, we revisit encoder-decoder LLM (RedLLM), enhancing it with recent recipes from decoder-only LLM (DecLLM). We conduct a comprehensive comparison between RedLLM, pretrained with prefix language modeling (LM), and DecLLM, pretrained with causal LM, at different model scales, ranging from $\sim$150M to $\sim$8B. Using RedPajama V1 (1.6T tokens) for pretraining and FLAN for instruction tuning, our experiments show that RedLLM produces compelling scaling properties and surprisingly strong performance. While DecLLM is overall more compute-optimal during pretraining, RedLLM demonstrates comparable scaling and context length extrapolation capabilities. After instruction tuning, RedLLM achieves comparable and even better results on various downstream tasks while enjoying substantially better inference efficiency. We hope our findings could inspire more efforts on re-examining RedLLM, unlocking its potential for developing powerful and efficient LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。