arXiv:2409.18057cs.CV2024-09ECCV被引 4

用神经光场实现快速高保真头像生成,实时渲染速度达174帧/秒。

LightAvatar: Efficient Head Avatar as Dynamic Neural Light Field

  • 基于神经光场构建头像模型,单次前向传播即可生成图像
  • 在512×512分辨率下实测达到174.1帧/秒,显著快于现有方法
  • 适合移动端或资源受限设备部署,兼顾质量与效率

近期工作表明,基于参数化模型的神经辐射场(NeRF)已能从单目视频生成高质量逼真的头部虚拟形象。然而,由于NeRF依赖密集采样,导致渲染速度慢,难以在资源受限设备上应用。本文提出LightAvatar,首个基于神经光场(NeLF)的头部虚拟形象模型。该模型仅需3DMM参数和相机位姿,通过一次网络前向传播即可生成图像,无需网格或体渲染。为解决实时性与训练稳定性难题,设计专用网络结构以保持低浮点运算量,并采用基于知识蒸馏的训练策略,利用预训练模型生成大量伪数据辅助训练。引入变形场网络修正真实数据拟合误差,提升学习效果。大量实验表明,该方法在图像质量上达到新SOTA,且显著更快,在消费级显卡RTX3090上实现174.1 FPS(512×512分辨率),无定制优化。

原文摘要 · Abstract (English)

Recent works have shown that neural radiance fields (NeRFs) on top of parametric models have reached SOTA quality to build photorealistic head avatars from a monocular video. However, one major limitation of the NeRF-based avatars is the slow rendering speed due to the dense point sampling of NeRF, preventing them from broader utility on resource-constrained devices. We introduce LightAvatar, the first head avatar model based on neural light fields (NeLFs). LightAvatar renders an image from 3DMM parameters and a camera pose via a single network forward pass, without using mesh or volume rendering. The proposed approach, while being conceptually appealing, poses a significant challenge towards real-time efficiency and training stability. To resolve them, we introduce dedicated network designs to obtain proper representations for the NeLF model and maintain a low FLOPs budget. Meanwhile, we tap into a distillation-based training strategy that uses a pretrained avatar model as teacher to synthesize abundant pseudo data for training. A warping field network is introduced to correct the fitting error in the real data so that the model can learn better. Extensive experiments suggest that our method can achieve new SOTA image quality quantitatively or qualitatively, while being significantly faster than the counterparts, reporting 174.1 FPS (512x512 resolution) on a consumer-grade GPU (RTX3090) with no customized optimization.

头像生成神经光场实时渲染高效建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。