让视觉语言模型说出推理过程，检查其物理逻辑是否真实。
World Models in Words: Auditing Physical State-Transition Commitments in Vision-Language Models
- 要求模型输出状态变化全过程，而非仅答案
- 35%正确答案背后存在物理不成立的推理链
- 适合评估模型是否真懂物理，而非猜对答案
视觉语言模型（VLMs）被广泛用于回答物理场景问题，但现有评估仅关注最终答案，隐藏了模型是否识别正确物体、表征正确物理状态、预测合理变化或仅凭错误理由选对答案。本文提出 \\(wmw\\)，一个审计模型语言表达的物理承诺的评估框架。不再只评分 $I,q\mapsto a$，而是要求模型生成带类型的推理轨迹 $I,q\mapsto(s_0,Δs,s_1,a)$：初始状态、状态变化、结果状态和答案。通过混合验证器检查结构合法性、状态锚定、变化一致性及答案与轨迹的匹配性，生成如物体、关系、力、变化、时间、单位/尺度、忠实性等错误标签。我们发布 \tracebank，包含 \nSeed 个经结构与重计算验证的合成场景，覆盖 \nFamilies 个物理类别，\nPairs 组微扰对比偏好对，验证代码、审计指南及模型输出。在控制与外部物理推理样本上评估 \nModels 个 VLMs。\wmw 揭示出答案评价忽略的失败：中等水平模型 35% 的正确答案对应物理无效的轨迹。验证器引导重排序提升最多 7 个百分点的轨迹有效性，且不牺牲答案准确率；轨迹级偏好调优使隐含不一致减少 41%。贡献不在于新基准，而在于可复用的协议，用于检验模型声称的物理世界是否与其答案同时为真。
原文摘要 · Abstract (English)
Vision-language models (VLMs) are increasingly used to answer questions about physical scenes, yet most evaluations reduce performance to a final answer. This hides whether the model perceived the right objects, represented the right physical state, predicted a plausible transition, or merely selected the right option for the wrong reasons. We introduce \wmw, an evaluation framework for auditing the \emph{language-expressed physical commitments} of VLMs. Instead of scoring only $I,q\mapsto a$, we ask models to produce a typed trace $I,q\mapsto(s_0,Δs,s_1,a)$: an initial state, a state transition, a resulting state, and an answer. A hybrid verifier then checks schema validity, state grounding, transition consistency, and answer-trace compatibility, yielding typed error labels such as object, relation, force, transition, temporal, unit/scale, and faithfulness errors. We release \tracebank, a controlled trace resource with \nSeed schema- and recomputation-validated synthetic scenarios across \nFamilies physics families, \nPairs minimally perturbed contrastive preference pairs, verifier code, audit guidelines, and model outputs. We evaluate \nModels VLMs on both controlled and external physical-reasoning examples. \wmw reveals failures that answer-only evaluation misses: 35\% of correct answers from mid-tier models are backed by physically invalid traces. Verifier-guided reranking recovers up to 7 percentage points of trace validity without sacrificing answer accuracy, and trace-level preference tuning reduces hidden inconsistency by 41\% relative. The contribution is not another final-answer physics benchmark, but a reusable protocol for measuring whether a VLM's stated physical world can be true at the same time as its answer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。