改进大模型世界模型构建，减少幻觉并提升一致性
Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement
- 用多标记预测增强表征结构，通过梯度耦合促进状态收敛
- 在合成图与真实网约车数据上显著降低结构幻觉，提升对扰动的鲁棒性
- 适合研究大模型内部表征、世界模型与生成一致性的学者
大语言模型是否具备连贯的内部世界模型仍是核心争议。传统单标记预测（NTP）仅关注一步前瞻监督，而多标记预测（MTP）展现出学习更结构化表征的潜力。本文从理论角度分析MTP的梯度归纳偏置，实证表明其通过梯度耦合诱导表征收缩，促进内部信念状态的收敛。然而，我们发现标准MTP常出现结构幻觉，即离散标记监督导致潜在空间中违反环境约束的非法捷径。为此，提出新方法LSE-MTP，将预测锚定于真实隐状态轨迹。在合成图和真实曼哈顿网约车数据集上的实验显示，LSE-MTP有效弥合离散标记与连续状态表示间的差距，提升表征对齐度，减少结构幻觉，并增强对扰动的鲁棒性。
原文摘要 · Abstract (English)
Whether Large Language Models (LLMs) develop coherent internal world models remains a core debate. While conventional Next-Token Prediction (NTP) focuses on one-step-ahead supervision, Multi-Token Prediction (MTP) has shown promise in learning more structured representations. In this work, we provide a theoretical perspective analyzing the gradient inductive bias of MTP, supported by empirical evidence, showing that MTP promotes the convergence toward internal belief states by inducing representational contractivity via gradient coupling. However, we reveal that standard MTP often suffers from structural hallucinations, where discrete token supervision encourages illegal shortcuts in latent space that violate environmental constraints. To address this, we propose a novel method Latent Semantic Enhancement MTP (LSE-MTP), which anchors predictions to ground-truth hidden state trajectories. Experiments on synthetic graphs and real-world Manhattan Taxi Ride show that LSE-MTP effectively bridges the gap between discrete tokens and continuous state representations, enhancing representation alignment, reducing structural hallucinations, and improving robustness to perturbations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。