让自回归模型学会补全被遮蔽的词，兼顾速度与效果。
Enabling Autoregressive Models to Fill In Masked Tokens
- 用线性解码器融合预训练的自回归与掩码语言模型的隐藏状态
- 在掩码补全任务上显著优于离散扩散模型等现有方法
- 保持自回归模型推理快、支持缓存的优势，适合实际部署
传统大模型通常采用自回归（AR）或掩码语言建模（MLM）目标训练，近年来自回归模型占据主导地位。但自回归模型无法处理上下文之间的遮蔽词补全任务，而掩码模型则存在训练和推理效率低的问题。本文提出MARIA（Masked and Autoregressive Infilling Architecture），通过将预训练的MLM与AR模型的隐藏状态拼接后输入线性解码器，实现对两者优势的结合。该方法仅需少量修改，使自回归模型具备补全能力，同时保留了其使用键值缓存带来的快速推理优势。实验表明，MARIA在掩码补全任务上的表现显著优于现有的离散扩散模型等方法。
原文摘要 · Abstract (English)
Historically, LLMs have been trained using either autoregressive (AR) or masked language modeling (MLM) objectives, with AR models gaining dominance in recent years. However, AR models are inherently incapable of masked infilling, which is the ability to predict masked tokens between past and future context. In contrast, MLM models suffer from intrinsic computational inefficiencies during both training and inference that hinder their scalability. This work introduces MARIA (Masked and Autoregressive Infilling Architecture), a novel approach that leverages the strengths of both paradigms to achieve state-of-the-art masked infilling performance. MARIA combines a pre-trained MLM and AR model by training a linear decoder that takes their concatenated hidden states as input. This minimal modification enables the AR model to perform infilling while retaining its inherent advantages in terms of faster inference with KV caching. Our results demonstrate that MARIA significantly outperforms existing methods, namely discrete diffusion models, on masked infilling tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。