arXiv:2412.09545cs.CVcs.GR2024-12CVPR被引 13

用文本生成可模拟的带发丝和衣物的3D人物,支持物理动画。

SimAvatar: Simulation-Ready Avatars with Layered Hair and Clothing

  • 分两阶段生成:先用3D高斯模型建模身体、衣物和发丝,再优化外观。
  • 生成的衣物和发丝可直接接入物理或神经模拟器,实现真实动态。
  • 适合需要高保真角色动画的影视、游戏与虚拟人场景。

我们提出SimAvatar,一个从文本提示生成可模拟的穿衣3D人体角色的框架。现有文本驱动人体生成方法要么统一建模头发、衣物和身体几何,要么生成的头发和衣物难以适配现有模拟流程。核心挑战在于如何表示头发和衣物几何,使其既能利用基础图像扩散模型(如Stable Diffusion)的先验知识,又可被物理或神经模拟器使用。为此,我们设计了两阶段框架,结合3D高斯的灵活性与可模拟的发丝和衣物网格。首先,使用三个文本条件的3D生成模型从文本提示生成衣物网格、身体形状和发丝。为利用扩散模型先验,我们在身体、衣物网格及发丝上附着3D高斯,并通过优化学习角色外观。给定姿态序列时,先对衣物网格和发丝应用物理模拟器,再通过针对各身体部位的定制机制将运动传递至3D高斯。结果是合成角色具有生动纹理和真实动态运动。据我们所知,该方法是首个生成高度逼真、完全可模拟的3D角色,超越当前主流方法。

原文摘要 · Abstract (English)

We introduce SimAvatar, a framework designed to generate simulation-ready clothed 3D human avatars from a text prompt. Current text-driven human avatar generation methods either model hair, clothing, and the human body using a unified geometry or produce hair and garments that are not easily adaptable for simulation within existing simulation pipelines. The primary challenge lies in representing the hair and garment geometry in a way that allows leveraging established prior knowledge from foundational image diffusion models (e.g., Stable Diffusion) while being simulation-ready using either physics or neural simulators. To address this task, we propose a two-stage framework that combines the flexibility of 3D Gaussians with simulation-ready hair strands and garment meshes. Specifically, we first employ three text-conditioned 3D generative models to generate garment mesh, body shape and hair strands from the given text prompt. To leverage prior knowledge from foundational diffusion models, we attach 3D Gaussians to the body mesh, garment mesh, as well as hair strands and learn the avatar appearance through optimization. To drive the avatar given a pose sequence, we first apply physics simulators onto the garment meshes and hair strands. We then transfer the motion onto 3D Gaussians through carefully designed mechanisms for each body part. As a result, our synthesized avatars have vivid texture and realistic dynamic motion. To the best of our knowledge, our method is the first to produce highly realistic, fully simulation-ready 3D avatars, surpassing the capabilities of current approaches.

3D生成角色模拟文本生成高斯

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。