发现语言指令会泄漏到模型预测中，导致看似懂空间关系实则只是抄指令。
Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix
- 用指令名称直接预测目标位置，而非真正感知场景。
- 去除指令后准确率从90%暴跌至27%，说明依赖指令而非视觉输入。
- 提出新训练方式：让模型读取世界状态，而非受指令干扰，恢复真实理解能力。
紧凑的世界模型通过语言目标条件化，试图用少量参考锚点来理解如“将红块放在蓝块左侧”这类空间关系。我们发现，看似高精度的0.90关系识别实际上只是指令转录，而非真实感知。当隐藏目标指令时，准确率骤降至0.27（三组种子），而使用反事实指令时，模型仍以94.5%概率预测错误位置（真实场景仅2.3%），表明存在指令泄漏。这一现象在三个设置及任务内消融中均成立。核心结论是：当评分量可直接从指令中推断出（如指令命名答案）时，便出现指令泄漏，且与非指令输入的预测能力无关。桌面任务和BabyAI基准存在此问题，而语言-表前向动力学模型若不命名方向则无泄漏；动作退化不会增加泄漏，与预测竞争理论相反。诊断建议：将目标从动态模型中移除（归于规划器成本），仅监督‘读取’路径，即可恢复真正的、不受指令影响的语义接地（0.88，有无目标结果一致）。该检测与修复方法适用于所有指令中包含评分项的目标条件模型。
原文摘要 · Abstract (English)
Compact world models that condition on a language goal promise to ground relations such as ``put the red block left of the blue block'' using a sparse set of explicit \emph{reference anchors}. We ask when such references actually ground a relation, and identify a trap: a goal-conditioned predictor reaches a striking $0.90$ relation-readout accuracy, yet this is \emph{instruction transcription}, not perception. Withholding the goal collapses it to chance ($0.90\!\to\!0.27$, three seeds) and a counterfactual instruction makes the predicted anchors follow the \emph{false} instruction $94.5\%$ of the time (true scene $2.3\%$; $N{=}256$). Tested across three settings and a within-task ablation, our central claim characterizes the confound: \textbf{instruction leakage occurs when the scored quantity is transcribable from the instruction (when the instruction names the answer) and is essentially independent of how predictive the non-instruction inputs are.} Our tabletop and the external BabyAI benchmark leak, whereas a Language-Table forward-dynamics world model whose instruction names \emph{referents} does not, until the instruction is augmented to name the direction; and degrading the action never increases leakage, the opposite of what predictor-competition predicts. The diagnosis prescribes the fix: keep the goal out of the dynamics (it belongs to the planner's cost) and supervise the \emph{read} path, recovering genuine, instruction-independent grounding ($0.88$, identical with and without the goal). The detection protocol and remedy apply to any goal-conditioned world model whose instruction names the scored quantity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。