用大模型+物理引擎自动推断物体参数,构建更真实的数字孪生。
LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Engines
- 通过大模型生成代码并驱动物理引擎,迭代优化参数与场景布局。
- 在新数据集上参数恢复精度超现有方法,收敛更稳定。
- 适合需要真实物理参数的机器人、自动驾驶等应用领域。
当前基于学习的复杂物理推理方法普遍忽略对决定场景动态的关键参数(如质量、摩擦系数)进行识别的问题,而这在碰撞避免、机器人操作等实际应用中至关重要。本文提出LLMPhy,一种将大语言模型(LLMs)与物理引擎结合的黑箱优化框架,用于物理推理。核心思想是将大模型中蕴含的教科书式物理知识与现代物理引擎中的世界模型相衔接,通过潜在参数估计构建输入场景的数字孪生。具体地,将数字孪生构建分解为连续的参数估计和离散的场景布局估计两个子问题。针对每个问题,LLMPhy迭代调用大模型生成编码参数估计的程序,于物理引擎中执行以重建场景,并利用重建误差作为反馈来优化大模型预测。由于现有物理推理基准很少考虑参数可识别性,我们引入三个新数据集,用于评估零样本条件下的物理推理能力。实验结果表明,LLMPhy在任务中达到当前最优表现,参数恢复更准确,收敛更可靠。
原文摘要 · Abstract (English)
Most learning-based approaches to complex physical reasoning sidestep the crucial problem of parameter identification (e.g., mass, friction) that governs scene dynamics, despite its importance in real-world applications such as collision avoidance and robotic manipulation. In this paper, we present LLMPhy, a black-box optimization framework that integrates large language models (LLMs) with physics simulators for physical reasoning. The core insight of LLMPhy is to bridge the textbook physical knowledge embedded in LLMs with the world models implemented in modern physics engines, enabling the construction of digital twins of input scenes via latent parameter estimation. Specifically, LLMPhy decomposes digital twin construction into two subproblems: (i) a continuous problem of estimating physical parameters and (ii) a discrete problem of estimating scene layout. For each subproblem, LLMPhy iteratively prompts the LLM to generate computer programs encoding parameter estimates, executes them in the physics engine to reconstruct the scene, and uses the resulting reconstruction error as feedback to refine the LLM's predictions. As existing physical reasoning benchmarks rarely account for parameter identifiability, we introduce three new datasets designed to evaluate physical reasoning in zero-shot settings. Our results show that LLMPhy achieves state-of-the-art performance on our tasks, recovers physical parameters more accurately, and converges more reliably than prior black-box methods. See the LLMPhy project page for details: https://www.merl.com/research/highlights/LLMPhy
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。