让视觉模拟与抽象推理协同,提升预测准确性与鲁棒性
World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning

- 用真实未来视频指导模型学习何时、如何使用视觉模拟
- 在两个新基准上分别提升10.6%和10.9%准确率
- 适合需要融合具象与抽象推理的AI系统开发者
世界模型和多模态大语言模型(MLLM)在从静态视觉观测中预测未来结果方面具有互补能力:世界模型可生成可能未来的具体视觉回溯,而MLLM能对问题、目标和规则进行抽象推理。然而,生成的回溯具有随机性,可能视觉合理但任务错误,因此需判断视觉模拟是否有效、回溯是否可信,以及如何影响最终答案。本文将此问题建模为受控的具象推理,提出一种模型学习调用、验证并整合视觉未来模拟与抽象推理。为此构建两个人工验证的基准:用于可控空间前瞻的VRQABench和用于开放域物理预测的OpenWorldQA。提出特权未来在线策略自蒸馏(PF-OPSD)方法:训练时利用真实未来视频和答案作为教师端特权信息评估在线具象推理轨迹,部署的学生模型在测试时从不观察真实未来。实验表明,PF-OPSD在VRQABench和OpenWorldQA上分别优于基线10.6%和10.9%,且对噪声或冲突回溯更具鲁棒性。代码与数据集已开源。
原文摘要 · Abstract (English)
World models and multimodal large language models (MLLMs) provide complementary capabilities for predicting future outcomes from static visual observations. World models can generate concrete visual rollouts of possible futures, while MLLMs can reason abstractly over questions, goals, and rules. However, generated rollouts are stochastic and may be visually plausible but task-incorrect, making it necessary to determine when visual simulation is useful, whether a rollout is credible, and how it should influence the final answer. We formulate this problem as controlled concrete reasoning, where a model learns to invoke, verify, and integrate visual future simulation alongside abstract reasoning. To study this setting, we construct two human-verified benchmarks, VRQABench for controllable spatial lookahead and OpenWorldQA for open-domain physical prediction, and propose Privileged-Future On-Policy Self-Distillation (PF-OPSD). During training, PF-OPSD uses ground-truth future videos and answers only as teacher-side privileged context to evaluate on-policy concrete-reasoning trajectories, while the deployable student never observes true futures at test time. Experimental results show that PF-OPSD outperforms baseline by 10.6% and 10.9% on VRQABench and OpenWorldQA, respectively, while increasing robustness to noisy or conflicting rollouts. Our code and dataset are available at https://github.com/yczhou001/PF-OPSD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。