用3D高斯点云实现实时逼真全身体感虚拟人,支持手机端流畅运行。
TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting
- 基于3D高斯点云构建可绑定的个性化人体模板,实现精细外观建模。
- 通过知识蒸馏将复杂形变网络压缩为轻量MLP,保持高频细节并实现实时渲染。
- 在Apple Vision Pro上稳定保持90帧/秒,适合AR直播与全息通信场景。
真实感3D全身体感虚拟人在增强现实领域具有广泛应用前景,涵盖电商直播到全息通信。尽管3D高斯点云(3DGS)在逼真虚拟人生成方面取得进展,现有方法在全身体感任务中仍难以精细控制面部表情与身体动作,且细节不足,无法在移动设备上实时运行。本文提出TaoAvatar,一种基于3DGS的高保真、轻量化全身体感虚拟人系统,支持多种信号驱动。首先构建个性化的穿衣服人体参数化模板,并将高斯点绑定以表示外观。接着预训练基于StyleUnet的网络处理复杂的姿态依赖非刚性形变,可捕捉高频外观细节,但计算开销过大。为此,我们采用知识蒸馏技术将非刚性形变“烘焙”至轻量级MLP网络,并设计混合变形形状补偿细节损失。大量实验表明,TaoAvatar在各类设备上均达到当前最优渲染质量,实现实时运行,在如Apple Vision Pro等高清立体设备上维持90帧/秒。
原文摘要 · Abstract (English)
Realistic 3D full-body talking avatars hold great potential in AR, with applications ranging from e-commerce live streaming to holographic communication. Despite advances in 3D Gaussian Splatting (3DGS) for lifelike avatar creation, existing methods struggle with fine-grained control of facial expressions and body movements in full-body talking tasks. Additionally, they often lack sufficient details and cannot run in real-time on mobile devices. We present TaoAvatar, a high-fidelity, lightweight, 3DGS-based full-body talking avatar driven by various signals. Our approach starts by creating a personalized clothed human parametric template that binds Gaussians to represent appearances. We then pre-train a StyleUnet-based network to handle complex pose-dependent non-rigid deformation, which can capture high-frequency appearance details but is too resource-intensive for mobile devices. To overcome this, we "bake" the non-rigid deformations into a lightweight MLP-based network using a distillation technique and develop blend shapes to compensate for details. Extensive experiments show that TaoAvatar achieves state-of-the-art rendering quality while running in real-time across various devices, maintaining 90 FPS on high-definition stereo devices such as the Apple Vision Pro.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。