将掩码与因果语言建模结合，打造可灵活切换的统一模型
GPT or BERT: why not both?
- 用混合训练目标融合掩码与因果建模
- 在BabyLM挑战中表现优于单一模式模型
- 适合需要双向与单向能力的下游任务
我们提出一种简单方法，将掩码语言建模与因果语言建模相结合。这种混合训练目标使模型在单一Transformer结构中同时具备两种范式的优点：GPT-BERT可像标准因果或掩码语言模型一样透明使用。我们在BabyLM Challenge 2024上测试了该预训练过程，结果表明混合预训练性能优于仅掩码或仅因果模型。我们公开发布模型、训练语料和代码。
原文摘要 · Abstract (English)
We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both modeling paradigms within a single transformer stack: GPT-BERT can be transparently used like any standard causal or masked language model. We test the pretraining process that enables this flexible behavior on the BabyLM Challenge 2024. The results show that the hybrid pretraining outperforms masked-only or causal-only models. We openly release the models, training corpora and code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。