用N元语法预测提升语言模型全局理解能力
N-gram Prediction and Word Difference Representations for Language Modeling
- 设计N元语法预测框架，让模型预判未来多个词
- 引入词差表示作为上下文感知的目标表征，增强语义捕捉
- 适合追求长距离依赖建模的自然语言处理研究者
因果语言建模（CLM）是当前大型语言模型取得成功的基础框架。尽管成效显著，但传统的下一个词预测训练方式可能导致模型过度关注句内局部依赖。虽然已有研究尝试同时预测未来N个词，但主要应用于掩码语言建模（MLM）和神经机器翻译（NMT）任务。本文提出一种适用于CLM任务的简单N-gram预测框架，并在该框架基础上引入词差表示（WDR）作为训练时的替代性、上下文感知的目标表征。为进一步提升下一个词预测质量，我们提出一种集成方法，融合未来N个词的预测结果。在涵盖CLM与NMT任务的多个基准数据集上的实证评估表明，所提方法显著优于传统CLM。
原文摘要 · Abstract (English)
Causal language modeling (CLM) serves as the foundational framework underpinning remarkable successes of recent large language models (LLMs). Despite its success, the training approach for next word prediction poses a potential risk of causing the model to overly focus on local dependencies within a sentence. While prior studies have been introduced to predict future N words simultaneously, they were primarily applied to tasks such as masked language modeling (MLM) and neural machine translation (NMT). In this study, we introduce a simple N-gram prediction framework for the CLM task. Moreover, we introduce word difference representation (WDR) as a surrogate and contextualized target representation during model training on the basis of N-gram prediction framework. To further enhance the quality of next word prediction, we propose an ensemble method that incorporates the future N words' prediction results. Empirical evaluations across multiple benchmark datasets encompassing CLM and NMT tasks demonstrate the significant advantages of our proposed methods over the conventional CLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。