通过语义规划提升语言模型训练，解决传统方法的捷径学习问题。
Semformer: Transformer Language Models with Semantic Planning
- 引入规划令牌序列，让模型预先预测响应的语义表示。
- 在图路径查找任务中实现接近完美的性能，有效避免捷径学习。
- 125M参数模型可直接预训练，适用于摘要等下游任务。
当前神经语言模型主要依赖下一个词预测，训练时采用教师强制策略，基于所有前序真实词元进行预测。然而，这种方法易产生捷径，利用已知前缀虚假拟合未来词元，可能损害预测准确性。本文提出Semformer，一种新型Transformer语言模型训练方法，显式建模响应的语义规划过程。具体地，在前缀中加入一系列规划令牌，引导其预测由自编码器诱导的响应潜在语义表示。在最小规划任务（即图路径查找）中，模型表现近乎完美，并有效缓解了捷径学习问题，而标准训练方法和基线模型未能达成此效果。此外，我们从零开始用125M参数预训练Semformer，通过困惑度、上下文学习及摘要任务微调等指标验证其有效性。
原文摘要 · Abstract (English)
Next-token prediction serves as the dominant component in current neural language models. During the training phase, the model employs teacher forcing, which predicts tokens based on all preceding ground truth tokens. However, this approach has been found to create shortcuts, utilizing the revealed prefix to spuriously fit future tokens, potentially compromising the accuracy of the next-token predictor. In this paper, we introduce Semformer, a novel method of training a Transformer language model that explicitly models the semantic planning of response. Specifically, we incorporate a sequence of planning tokens into the prefix, guiding the planning token representations to predict the latent semantic representations of the response, which are induced by an autoencoder. In a minimal planning task (i.e., graph path-finding), our model exhibits near-perfect performance and effectively mitigates shortcut learning, a feat that standard training methods and baseline models have been unable to accomplish. Furthermore, we pretrain Semformer from scratch with 125M parameters, demonstrating its efficacy through measures of perplexity, in-context learning, and fine-tuning on summarization tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。