用粒子与网格结合建模可变形物体,从少量视觉数据中学习动态行为。
Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos
- 粒子表示形状,网格保持空间连续性,提升学习效率。
- 在稀疏视角下仍能准确模拟绳子、布料等多类物体动态。
- 适合做基于模型的机器人规划,支持任务导向操控。
由于可变形物体物理特性多样且状态估计受限于视觉信息,其动力学建模极具挑战。本文提出一种结合粒子与空间网格的混合表示神经动力学框架,通过粒子捕捉全局形变与运动,利用空间网格保证三维空间连续性并提升学习效率。该方法融合高斯溅射实现可视化渲染,构建完全学习驱动的可变形物体数字孪生,并生成3D动作条件视频。实验表明,模型仅需稀疏视角的RGB-D数据即可学习绳索、布料、填充玩具和纸袋等多样化物体的动力学,且具备类别级泛化能力。相比最先进学习与物理基模拟器,在少视角场景下表现更优。此外,所学模型可用于基于模型的规划,支持多种任务的目标导向操作。项目页面:https://kywind.github.io/pgnd。
原文摘要 · Abstract (English)
Modeling the dynamics of deformable objects is challenging due to their diverse physical properties and the difficulty of estimating states from limited visual information. We address these challenges with a neural dynamics framework that combines object particles and spatial grids in a hybrid representation. Our particle-grid model captures global shape and motion information while predicting dense particle movements, enabling the modeling of objects with varied shapes and materials. Particles represent object shapes, while the spatial grid discretizes the 3D space to ensure spatial continuity and enhance learning efficiency. Coupled with Gaussian Splattings for visual rendering, our framework achieves a fully learning-based digital twin of deformable objects and generates 3D action-conditioned videos. Through experiments, we demonstrate that our model learns the dynamics of diverse objects -- such as ropes, cloths, stuffed animals, and paper bags -- from sparse-view RGB-D recordings of robot-object interactions, while also generalizing at the category level to unseen instances. Our approach outperforms state-of-the-art learning-based and physics-based simulators, particularly in scenarios with limited camera views. Furthermore, we showcase the utility of our learned models in model-based planning, enabling goal-conditioned object manipulation across a range of tasks. The project page is available at https://kywind.github.io/pgnd .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。