用倒数第二字预测提升语言模型生成质量
Improving Next Tokens via Second-to-Last Predictions with Generate and Refine
- 训练解码器预测序列倒数第二字，结构化遮蔽提升效率
- 倒数第二字预测准确率比常规预测高15%以上
- 结合生成与修正策略，显著优化下游文本生成
自回归语言模型（如GPT）旨在预测下一个词元，而自编码模型（如BERT）则通过掩码词元预测任务进行训练。本文训练一个仅解码器架构，用于预测序列中倒数第二个词元。该方法通过结构化的确定性遮蔽方式，在训练效率上优于BERT型模型。利用该模型，我们提出“生成-修正”框架，融合倒数第二词元预测与标准的下一个词元预测，以改进GPT类模型的生成效果。在多个GPT-2变体和不同数据集上的实验表明，倒数第二词元预测准确率显著更高，较标准方法提升超过15%。‘生成-修正’策略亦带来明显但稳定且显著的改进，验证了其有效性。
原文摘要 · Abstract (English)
Autoregressive language models like GPT aim to predict next tokens, while autoencoding models such as BERT are trained on tasks such as predicting masked tokens. We train a decoder-only architecture for predicting the second to last token for a sequence of tokens. Our approach yields higher computational training efficiency than BERT-style models by employing a structured deterministic approach to masking tokens. We use our model to improve the next token predictions of a standard GPT by combining both predictions in a ``generate-then-refine'' approach. We demonstrate on different variants of GPT-2 and different datasets that (not unexpectedly) second to last token predictions are much more accurate, i.e., more than 15\% higher accuracy than standard next token predictions. The ``generate-then-refine'' approach also demonstrates notable improvements in next-token predictions, yielding smaller yet consistent and significant gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。