将物理规律融入视觉推理,让模型像人一样理解动态世界。
SlotPi: Physics-informed Object-centric Reasoning Models
- 用哈密顿原理构建物理模块,结合时空预测实现对象动态推理。
- 在基准数据集和流体数据集上,预测与视觉问答任务均表现优异。
- 支持真实世界复杂场景,适合需要物理常识的智能系统研发。
通过视觉观察理解并推理受物理定律支配的动态过程,类似于人类在现实世界中的能力,仍面临巨大挑战。当前的对象中心动态模拟方法虽取得显著进展,但忽略了两个关键方面:一是将物理知识融入模型;二是验证模型在多样化场景下的适应性。现实世界中的动态,尤其是涉及流体与物体的交互,要求模型不仅能捕捉对象间相互作用,还需模拟流体流动特性。为此,我们提出SlotPi——一种基于槽(slot)的物理信息对象中心推理模型。该模型融合基于哈密顿原理的物理模块与时空预测模块,用于动态预测。实验表明,模型在基准数据集和流体数据集上的预测与视觉问答(VQA)任务中表现优异。此外,我们构建了一个包含对象交互、流体动力学及流体-对象交互的真实世界数据集,并在此上验证了模型性能。其在所有数据集上的稳健表现,彰显了强大的泛化能力,为构建更先进的世界模型奠定了基础。
原文摘要 · Abstract (English)
Understanding and reasoning about dynamics governed by physical laws through visual observation, akin to human capabilities in the real world, poses significant challenges. Currently, object-centric dynamic simulation methods, which emulate human behavior, have achieved notable progress but overlook two critical aspects: 1) the integration of physical knowledge into models. Humans gain physical insights by observing the world and apply this knowledge to accurately reason about various dynamic scenarios; 2) the validation of model adaptability across diverse scenarios. Real-world dynamics, especially those involving fluids and objects, demand models that not only capture object interactions but also simulate fluid flow characteristics. To address these gaps, we introduce SlotPi, a slot-based physics-informed object-centric reasoning model. SlotPi integrates a physical module based on Hamiltonian principles with a spatio-temporal prediction module for dynamic forecasting. Our experiments highlight the model's strengths in tasks such as prediction and Visual Question Answering (VQA) on benchmark and fluid datasets. Furthermore, we have created a real-world dataset encompassing object interactions, fluid dynamics, and fluid-object interactions, on which we validated our model's capabilities. The model's robust performance across all datasets underscores its strong adaptability, laying a foundation for developing more advanced world models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。