双向预测模型提升文本生成的推理能力，解决传统模型难以处理的复杂任务。
The Belief State Transformer
- 输入前缀和后缀，同时预测下一个词和上一个词，构建紧凑信念状态。
- 在已知目标时超越填空法，在未知目标时仍表现更优，提升推理效率。
- 适用于需要精准控制生成结果的场景，如故事续写、逻辑推理等。
我们提出「信念状态变换器」（Belief State Transformer），一种同时以前缀和后缀为输入的下一个词预测模型，其新颖目标是同时预测前缀的下一个词和后缀的上一个词。该模型以领域无关的方式有效解决传统单向变换器难以应对的挑战性问题，关键在于学习一个紧凑的信念状态，捕获准确预测所需的所有相关信息。实验消融表明，在标准变换器失效的复杂场景中，模型各组件均至关重要。在已知前缀与后缀的故事创作任务中，该方法在达成已知目标方面优于填空中间法，并在目标未知时也展现出更优性能。整体而言，信念状态变换器实现了更高效的条件解码、更优的测试时推断以及小规模问题上的高质量文本表征。
原文摘要 · Abstract (English)
We introduce the "Belief State Transformer", a next-token predictor that takes both a prefix and suffix as inputs, with a novel objective of predicting both the next token for the prefix and the previous token for the suffix. The Belief State Transformer effectively learns to solve challenging problems that conventional forward-only transformers struggle with, in a domain-independent fashion. Key to this success is learning a compact belief state that captures all relevant information necessary for accurate predictions. Empirical ablations show that each component of the model is essential in difficult scenarios where standard Transformers fall short. For the task of story writing with known prefixes and suffixes, our approach outperforms the Fill-in-the-Middle method for reaching known goals and demonstrates improved performance even when the goals are unknown. Altogether, the Belief State Transformer enables more efficient goal-conditioned decoding, better test-time inference, and high-quality text representations on small scale problems. Website: https://edwhu.github.io/bst-website
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。