AntLM融合因果与掩码语言建模，提升训练效率与性能
AntLM: Bridging Causal and Masked Language Models
- 交替使用因果与双向注意力及对应目标函数
- 在相同训练轮次下，宏平均指标提升1%至2.2%
- 适合追求高效预训练的模型开发者参考
因果语言建模（CLM）和掩码语言建模（MLM）是基于Transformer网络的两种主流学习范式，分别对应仅解码器和仅编码器架构。二者在下游任务中各有优劣。在2023年BabyLM挑战赛中，尽管MLM平均表现最佳，但CLM收敛速度显著更快。针对2024年BabyLM挑战赛，我们提出一种新范式AntLM，融合CLM与MLM，以兼顾两者优势。实验选取严格小规模赛道，在两个基础模型上进行：代表CLM的BabyLlama和代表MLM的LTG-BERT。训练过程中交替应用CLM或MLM目标函数，以及因果或双向注意力掩码。结果表明，联合预训练目标能有效发挥各自优势，提升整体训练性能。相同训练轮次下，AntLM_{BabyLlama}宏平均提升1%，AntLM_{LTG-BERT}提升2.2%。
原文摘要 · Abstract (English)
Causal Language Modeling (CLM) and Masked Language Modeling (MLM) are two mainstream learning paradigms based on Transformer networks, specifically the Decoder-only and Encoder-only architectures. The strengths of each paradigm in downstream tasks have shown a mix of advantages and disadvantages. In the past BabyLM Challenge 2023, although the MLM paradigm achieved the best average performance, the CLM paradigm demonstrated significantly faster convergence rates. For the BabyLM Challenge 2024, we propose a novel language modeling paradigm named $\textbf{AntLM}$, which integrates both CLM and MLM to leverage the advantages of these two classic paradigms. We chose the strict-small track and conducted experiments on two foundation models: BabyLlama, representing CLM, and LTG-BERT, representing MLM. During the training process for specific foundation models, we alternate between applying CLM or MLM training objectives and causal or bidirectional attention masks. Experimental results show that combining the two pretraining objectives leverages their strengths, enhancing overall training performance. Under the same epochs, $AntLM_{BabyLlama}$ improves Macro-average by 1%, and $AntLM_{LTG-BERT}$ achieves a 2.2% increase over the baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。