让视觉模型通过互动学习物理直觉,无需重训练即可适应新环境。
ICPRL: Acquiring Physical Intuition from Interactive Control
- 基于多轮交互历史,用分组相对策略优化训练视觉策略模型。
- 在DeepPHY基准上,新环境下的任务成功率提升显著,且无需权重更新。
- 适合需要快速适应物理环境的机器人控制与具身智能研究者。
视觉语言模型(VLMs)在静态感知上表现优异,但在动态物理环境中进行交互推理时表现不足,这需要规划和对动态结果的适应能力。现有物理推理方法常依赖抽象符号输入,或无法从像素级视觉互动中学习并适应新场景。我们提出ICPRL(上下文物理强化学习),受上下文强化学习(ICRL)启发,使VLM能够通过上下文交互获得物理直觉并自适应策略。该方法通过多轮分组相对策略优化(GRPO)在多样化的多回合交互历史上训练视觉策略模型,使代理能基于过往试错序列调整策略,无需任何权重更新。该自适应策略与独立训练的世界模型协同工作,后者通过预测潜在动作的结果提供显式物理推理。推理时,策略生成候选动作,世界模型预测结果,引导根节点PUCT搜索选择最优动作。在DeepPHY基准的多样化物理谜题任务上,ICPRL在仅策略阶段和世界模型增强阶段均表现出显著提升,且在未见物理环境中仍保持性能,证明框架能真正从交互经验中获取环境物理动态的上下文知识。
原文摘要 · Abstract (English)
VLMs excel at static perception but falter in interactive reasoning in dynamic physical environments, which demands planning and adaptation to dynamic outcomes. Existing physical reasoning methods often depend on abstract symbolic inputs or lack the ability to learn and adapt from direct, pixel-based visual interaction in novel scenarios. We introduce ICPRL (In-Context Physical Reinforcement Learning), a framework inspired by In-Context Reinforcement Learning (ICRL) that empowers VLMs to acquire physical intuition and adapt their policies in-context. Our approach trains a vision-grounded policy model via multi-turn Group Relative Policy Optimization (GRPO) over diverse multi-episode interaction histories. This enables the agent to adapt strategies by conditioning on past trial-and-error sequences, without requiring any weight updates. This adaptive policy works in concert with a separately trained world model that provides explicit physical reasoning by predicting the results of potential actions. At inference, the policy proposes candidate actions, while the world model predicts outcomes to guide a root-node PUCT search to select the most promising action. Evaluated on the diverse physics-based puzzle-solving tasks in the DeepPHY benchmark, ICPRL demonstrates significant improvements across both its I. policy-only, and II. world-model-augmented stages. Notably, these gains are retained in unseen physical environments, demonstrating that our framework facilitates genuine in-context acquisition of the environment's physical dynamics from interactive experience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。