对比编码器与解码器模型性能,发现双向适配效果差于专用模型。
Seq vs Seq: An Open Suite of Paired Encoders and Decoders
- 构建了从1700万到10亿参数的成对编码器/解码器模型。
- 相同训练策略下,编码器在分类任务上优于解码器,反之亦然。
- 开源全部训练数据与200多个检查点,支持后续研究复现与扩展。
大语言模型社区主要使用仅解码器的模型进行文本生成,但仍有大量工作使用仅编码器模型处理分类或检索任务。以往研究因参数量、训练方法和数据集不一致而难以公平比较。本文推出SOTA开源的Ettin模型套件:包含从1700万到10亿参数的成对编码器与解码器模型,均在最多2万亿词上训练。采用统一训练方案,使两类模型在各自规模下达到最优表现,超越ModernBERT(编码器)与Llama 3.2、SmolLM2(解码器)。结果显示,编码器在分类与检索任务中占优,解码器在生成任务中更优;但通过持续训练将模型互转效果不佳——例如400万参数编码器在MNLI上优于10亿参数解码器,反之亦然。所有实验数据、训练顺序及200多个检查点均已开源,支持未来研究分析与扩展。
原文摘要 · Abstract (English)
The large language model (LLM) community focuses almost exclusively on decoder-only language models, since they are easier to use for text generation. However, a large subset of the community still uses encoder-only models for tasks such as classification or retrieval. Previous work has attempted to compare these architectures, but is forced to make comparisons with models that have different numbers of parameters, training techniques, and datasets. We introduce the SOTA open-data Ettin suite of models: paired encoder-only and decoder-only models ranging from 17 million parameters to 1 billion, trained on up to 2 trillion tokens. Using the same recipe for both encoder-only and decoder-only models produces SOTA recipes in both categories for their respective sizes, beating ModernBERT as an encoder and Llama 3.2 and SmolLM2 as decoders. Like previous work, we find that encoder-only models excel at classification and retrieval tasks while decoders excel at generative tasks. However, we show that adapting a decoder model to encoder tasks (and vice versa) through continued training is subpar compared to using only the reverse objective (i.e. a 400M encoder outperforms a 1B decoder on MNLI, and vice versa for generative tasks). We open-source all artifacts of this study including training data, training order segmented by checkpoint, and 200+ checkpoints to allow future work to analyze or extend all aspects of training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。