实时流中快速重建3D人脸形象,无需预存数据。
StreamME: Simplify 3D Gaussian Avatar within Live Stream
- 边采集边重建,用稀疏关键点优化点云分布
- 训练速度极快,支持表情快速适配
- 适合虚拟会议、动画制作等场景,保护隐私
我们提出StreamME,一种专注于快速3D人物形象重建的方法。该方法可同步接收并重建来自实时视频流的头部形象,无需任何预缓存数据,实现重建结果无缝融入下游应用。其核心在于‘即刻训练’策略,基于3D高斯溅射(3DGS),摒弃可变形3DGS中的MLP,仅依赖几何信息,显著提升对表情变化的适应速度。为进一步保障即刻训练的高效性,我们提出基于主点的简化策略,使点云在面部表面更稀疏分布,在控制点数的同时保持渲染质量。借助即刻训练能力,本方法可保护面部隐私,并降低虚拟现实系统或在线会议中的通信带宽。此外,可直接应用于动画、卡通化及再照明等下游任务。更多详情请访问项目页:https://songluchuan.github.io/StreamME/。
原文摘要 · Abstract (English)
We propose StreamME, a method focuses on fast 3D avatar reconstruction. The StreamME synchronously records and reconstructs a head avatar from live video streams without any pre-cached data, enabling seamless integration of the reconstructed appearance into downstream applications. This exceptionally fast training strategy, which we refer to as on-the-fly training, is central to our approach. Our method is built upon 3D Gaussian Splatting (3DGS), eliminating the reliance on MLPs in deformable 3DGS and relying solely on geometry, which significantly improves the adaptation speed to facial expression. To further ensure high efficiency in on-the-fly training, we introduced a simplification strategy based on primary points, which distributes the point clouds more sparsely across the facial surface, optimizing points number while maintaining rendering quality. Leveraging the on-the-fly training capabilities, our method protects the facial privacy and reduces communication bandwidth in VR system or online conference. Additionally, it can be directly applied to downstream application such as animation, toonify, and relighting. Please refer to our project page for more details: https://songluchuan.github.io/StreamME/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。