用单目视频生成可实时渲染的逼真虚拟人,直接兼容Unity引擎。
GSAC: Leveraging Gaussian Splatting for Photorealistic Avatar Creation with Unity Integration
- 基于3D高斯溅射,从普通视频一键生成高精度虚拟人。
- 支持面部表情细节还原,能在Unity中实时交互运行。
- 适合元宇宙、VR/AR开发者快速搭建逼真人像应用。
逼真的虚拟人对虚拟现实(VR)和增强现实(AR)中的沉浸式应用至关重要,广泛应用于训练模拟、远程医疗和虚拟协作等领域。现有技术存在成本高、制作周期长、难以实现实时渲染等问题:人工方法如MetaHuman耗时耗力,自动方法如基于NeRF的流程效率低、面部表情还原不充分,且无法满足实时性要求。为此,我们提出一个端到端的3D高斯溅射(3DGS)虚拟人创建流程,仅需单目视频输入,即可生成可直接在Unity引擎中使用的高效、逼真虚拟人。该流程引入定制化预处理机制,支持“野生”场景下的单目视频采集,实现精细的面部表情重建并嵌入完整绑定模型。同时,我们开发了集成于Unity的高斯溅射虚拟人编辑器,提供友好的开发环境。实验验证了预处理流程对自定义数据标准化的有效性,并展示了高斯虚拟人在Unity中的多功能性与实用性。
原文摘要 · Abstract (English)
Photorealistic avatars have become essential for immersive applications in virtual reality (VR) and augmented reality (AR), enabling lifelike interactions in areas such as training simulations, telemedicine, and virtual collaboration. These avatars bridge the gap between the physical and digital worlds, improving the user experience through realistic human representation. However, existing avatar creation techniques face significant challenges, including high costs, long creation times, and limited utility in virtual applications. Manual methods, such as MetaHuman, require extensive time and expertise, while automatic approaches, such as NeRF-based pipelines often lack efficiency, detailed facial expression fidelity, and are unable to be rendered at a speed sufficent for real-time applications. By involving several cutting-edge modern techniques, we introduce an end-to-end 3D Gaussian Splatting (3DGS) avatar creation pipeline that leverages monocular video input to create a scalable and efficient photorealistic avatar directly compatible with the Unity game engine. Our pipeline incorporates a novel Gaussian splatting technique with customized preprocessing that enables the user of "in the wild" monocular video capture, detailed facial expression reconstruction and embedding within a fully rigged avatar model. Additionally, we present a Unity-integrated Gaussian Splatting Avatar Editor, offering a user-friendly environment for VR/AR application development. Experimental results validate the effectiveness of our preprocessing pipeline in standardizing custom data for 3DGS training and demonstrate the versatility of Gaussian avatars in Unity, highlighting the scalability and practicality of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。