让人体与物体在3D中可动且逼真,支持新视角和新姿势的交互渲染。
GASPACHO: Gaussian Splatting for Controllable Humans and Objects
- 用2D表面流形学习物体高斯点,提升动态物体细节清晰度。
- 在高斯空间引入接触约束,实现自然物理合理的动作动画。
- 首次实现神经渲染下人体与动态物体在复杂场景中的可控交互生成。
我们提出GASPACHO,一种从多视角RGB视频生成逼真可控人体-物体交互渲染的方法。不同于以往仅重建人体而将物体视为背景的工作,GASPACHO同时恢复人体和交互物体的可动画模板,以独立的高斯点集表示,从而支持从新视角、新姿态生成新颖的人体-物体交互。我们提出一种新型公式,在底层2D表面流形上学习物体高斯点,而非3D体积,显著提升动态物体重建的锐度与细粒度。此外,我们设计了高斯空间中的接触约束,规范人体与物体的关系,实现自然且物理合理的动画。在BEHAVE、NeuralDome和DNA-Rendering三个基准上,GASPACHO在严重遮挡下仍能实现高质量重建,并支持新交互的可控合成。我们还展示了该方法可将人体与物体组合至3D场景中,首次证明神经渲染可用于多样场景中可控生成逼真人与动态物体的交互。结果详见:https://miraymen.github.io/gaspacho/
原文摘要 · Abstract (English)
We present GASPACHO, a method for generating photorealistic, controllable renderings of human-object interactions from multi-view RGB video. Unlike prior work that reconstructs only the human and treats objects as background, GASPACHO simultaneously recovers animatable templates for both the human and the interacting object as distinct sets of Gaussians, thereby allowing for controllable renderings of novel human object interactions in different poses from novel-camera viewpoints. We introduce a novel formulation that learns object Gaussians on an underlying 2D surface manifold rather than in 3D volume, yielding sharper, fine-grained object details for dynamic object reconstruction. We further propose a contact constraint in Gaussian space that regularizes human-object relations and enables natural, physically plausible animation. Across three benchmarks - BEHAVE, NeuralDome, and DNA-Rendering - GASPACHO achieves high-quality reconstructions under heavy occlusion and supports controllable synthesis of novel human-object interactions. We also demonstrate that our method allows for composition of humans and objects in 3D scenes and for the first time showcase that neural rendering can be used for the controllable generation of photoreal humans interacting with dynamic objects in diverse scenes. Our results are available at: https://miraymen.github.io/gaspacho/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。