用大模型自动推断物体物理属性,实现快速真实动态3D模拟
Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splatting
- 通过多模态大模型零样本预测物体平均物理属性
- 基于几何与分布估计,实现复杂形变模拟且计算成本降低
- 单卡2分钟内完成真实感动态生成,适合交互式3D应用
近期3D生成模型的发展为动态物体运动模拟与行为定制带来了新可能,但内容创建仍具挑战。现有方法通常需手动设定精确物理属性,或依赖视频生成模型推断,计算开销大。本文重新思考多模态大语言模型(MLLM)在物理仿真中的作用,提出Sim Anything:一种赋予静态3D物体交互动态的物理驱动方法。首先进行精细场景重建与物体级3D开放词汇分割,再执行多视角图像修复。受人类视觉推理启发,提出基于MLLM的物理属性感知(MLLM-P3),零样本预测物体平均物理属性。结合平均值与几何信息,材料属性分布预测模型(MPDP)将问题重构为概率分布估计,降低计算成本。最后,采用物理-几何自适应采样(PGAS)策略,在开放世界场景中以粒子模拟物体,高效捕捉复杂形变。大量实验与用户研究显示,Sim Anything在单个GPU上2分钟内即可实现优于当前最先进方法的真实运动表现。
原文摘要 · Abstract (English)
Recent advancements in 3D generation models have opened new possibilities for simulating dynamic 3D object movements and customizing behaviors, yet creating this content remains challenging. Current methods often require manual assignment of precise physical properties for simulations or rely on video generation models to predict them, which is computationally intensive. In this paper, we rethink the usage of multi-modal large language model (MLLM) in physics-based simulation, and present Sim Anything, a physics-based approach that endows static 3D objects with interactive dynamics. We begin with detailed scene reconstruction and object-level 3D open-vocabulary segmentation, progressing to multi-view image in-painting. Inspired by human visual reasoning, we propose MLLM-based Physical Property Perception (MLLM-P3) to predict mean physical properties of objects in a zero-shot manner. Based on the mean values and the object's geometry, the Material Property Distribution Prediction model (MPDP) model then estimates the full distribution, reformulating the problem as probability distribution estimation to reduce computational costs. Finally, we simulate objects in an open-world scene with particles sampled via the Physical-Geometric Adaptive Sampling (PGAS) strategy, efficiently capturing complex deformations and significantly reducing computational costs. Extensive experiments and user studies demonstrate our Sim Anything achieves more realistic motion than state-of-the-art methods within 2 minutes on a single GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。