将解码器模型适配为编码器，拓展其在分类等任务的应用
Adapting Decoder-Based Language Models for Diverse Encoder Downstream Tasks
- 将Gemma解码器转换为编码器结构，通过池化与注意力优化适配
- 在GLUE和MS MARCO上表现优于传统编码器模型，验证通用性
- 适合需要强大语言理解能力的非生成类任务研究者
尽管基于解码器的Transformer在语言建模中取得突破并可扩展至超大规模，但在自然语言处理中仍未能完全取代以编码器为主的架构。尤其在分类、回归和排序等任务上，编码器模型仍占主导地位。这主要源于解码器模型固有的结构限制，使其难以直接应用于此类任务。本文提出Gemma Encoder，将强大的Gemma解码器模型适配为编码器架构，从而释放其在多样化非生成任务中的潜力。为优化从解码器到编码器的转换，我们系统分析了多种池化策略、注意力机制及超参数（如丢弃率）。此外，在GLUE基准和MS MARCO排序基准上对Gemma Encoder进行了基准测试，结果证明其有效性与多功能性。
原文摘要 · Abstract (English)
Decoder-based transformers, while revolutionizing language modeling and scaling to immense sizes, have not completely overtaken encoder-heavy architectures in natural language processing. Specifically, encoder-only models remain dominant in tasks like classification, regression, and ranking. This is primarily due to the inherent structure of decoder-based models, which limits their direct applicability to these tasks. In this paper, we introduce Gemma Encoder, adapting the powerful Gemma decoder model to an encoder architecture, thereby unlocking its potential for a wider range of non-generative applications. To optimize the adaptation from decoder to encoder, we systematically analyze various pooling strategies, attention mechanisms, and hyperparameters (e.g., dropout rate). Furthermore, we benchmark Gemma Encoder against established approaches on the GLUE benchmarks, and MS MARCO ranking benchmark, demonstrating its effectiveness and versatility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。