arXiv:2412.05560cs.CVcs.AI2024-12被引 2

用大模型优化提示词,生成逼真且会按物理规律动的3D模型。

Text-to-3D Gaussian Splatting with Physics-Grounded Motion Generation

  • 用大模型改进文本提示,结合扩散先验指导高斯点云生成
  • 通过连续力学模型实现物体在受力下的真实形变与运动
  • 适合需要物理仿真效果的VR/游戏/动画制作人员

文本到3D生成是虚拟现实与数字内容创作中的关键技术。尽管近期研究已推动该领域发展,但高效提示下生成高保真3D物体,以及准确模拟其物理驱动运动仍面临挑战。为此,我们提出一个创新框架:利用大语言模型(LLM)优化提示词,并以扩散先验引导高斯点云(Gaussian Splatting, GS)生成具有精确外观与几何结构的3D模型。同时,引入基于连续力学的变形图与颜色正则化,合成生动的物理感知运动,满足质量与动量守恒。该框架将文本到3D生成与物理驱动运动融合,生成具有物理意识动态行为的逼真3D对象,能准确反映不同材料在各种力与约束下的表现。大量实验表明,本方法在高质量3D生成与真实物理运动方面均取得显著效果。

原文摘要 · Abstract (English)

Text-to-3D generation is a valuable technology in virtual reality and digital content creation. While recent works have pushed the boundaries of text-to-3D generation, producing high-fidelity 3D objects with inefficient prompts and simulating their physics-grounded motion accurately still remain unsolved challenges. To address these challenges, we present an innovative framework that utilizes the Large Language Model (LLM)-refined prompts and diffusion priors-guided Gaussian Splatting (GS) for generating 3D models with accurate appearances and geometric structures. We also incorporate a continuum mechanics-based deformation map and color regularization to synthesize vivid physics-grounded motion for the generated 3D Gaussians, adhering to the conservation of mass and momentum. By integrating text-to-3D generation with physics-grounded motion synthesis, our framework renders photo-realistic 3D objects that exhibit physics-aware motion, accurately reflecting the behaviors of the objects under various forces and constraints across different materials. Extensive experiments demonstrate that our approach achieves high-quality 3D generations with realistic physics-grounded motion.

3D生成物理模拟高斯点云文本生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。