让大模型具备物理预判能力,实时规划更可靠的行动策略。
APEX: Empowering LLMs with Physics-Based Task Planning for Real-time Insight
- 构建动态交互图与低延迟物理仿真,增强大模型的环境预见力。
- 在三类基准测试中显著优于传统大模型和视觉语言模型。
- 适合需要实时决策与物理交互的机器人、自动驾驶等场景。
大语言模型虽具强大推理与规划能力,但在物理交互建模上仍存在根本局限。现有方法通过视觉-语言模型感知或强化学习实现自适应决策,但难以捕捉动态物体交互,且常需任务特异性训练,限制了实际应用。我们提出APEX(前瞻性物理增强执行)框架,赋予大模型基于物理的预见能力以实现实时任务规划。APEX通过构建结构化图识别并建模环境中最关键的动态交互,为大模型提供明确的物理状态更新;同时,它支持低延迟的物理可行动作前向仿真,使大模型能依据预测结果而非静态观测选择最优策略。我们在三个评测基准上验证了APEX:(1) 物理推理基准,检验因果推断与物体运动预测能力;(2) Tetris,评估物理感知预测对长时程规划决策的提升效果;(3) 动态障碍物避障,考察感知与动作可行性分析的即时融合。实验表明,APEX显著优于标准大模型及基于视觉语言模型的方法,证明显式物理推理对弥合语言智能与真实世界任务执行之间鸿沟至关重要。源代码与实验配置已公开于https://github.com/hwj20/APEX_EXP。
原文摘要 · Abstract (English)
Large Language Models (LLMs) demonstrate strong reasoning and task planning capabilities but remain fundamentally limited in physical interaction modeling. Existing approaches integrate perception via Vision-Language Models (VLMs) or adaptive decision-making through Reinforcement Learning (RL), but they fail to capture dynamic object interactions or require task-specific training, limiting their real-world applicability. We introduce APEX (Anticipatory Physics-Enhanced Execution), a framework that equips LLMs with physics-driven foresight for real-time task planning. APEX constructs structured graphs to identify and model the most relevant dynamic interactions in the environment, providing LLMs with explicit physical state updates. Simultaneously, APEX provides low-latency forward simulations of physically feasible actions, allowing LLMs to select optimal strategies based on predictive outcomes rather than static observations. We evaluate APEX on three benchmarks designed to assess perception, prediction, and decision-making: (1) Physics Reasoning Benchmark, testing causal inference and object motion prediction; (2) Tetris, evaluating whether physics-informed prediction enhances decision-making performance in long-horizon planning tasks; (3) Dynamic Obstacle Avoidance, assessing the immediate integration of perception and action feasibility analysis. APEX significantly outperforms standard LLMs and VLM-based models, demonstrating the necessity of explicit physics reasoning for bridging the gap between language-based intelligence and real-world task execution. The source code and experiment setup are publicly available at https://github.com/hwj20/APEX_EXP .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。