arXiv:2410.00630cs.CVcs.AI2024-10SIGGRAPH被引 39

仅用三张随意拍摄的照片,就能还原逼真人脸表情与细节。

Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures

  • 用合成数据训练隐式人脸先验,再用真实少量照片微调。
  • 平均仅需三张输入图像,即可实现高质量新视角生成。
  • 适合低资源场景下的人脸3D建模,如短视频创作或虚拟形象生成。

体积分法和神经辐射场表示已革新3D人脸捕捉与逼真新视角生成。然而,这些方法通常需要数百张多视角输入图像,难以应对少样本情况。本文提出一种新型人脸体积分法先验,仅需最少三张野外采集的输入视图,即可实现高保真表达性人脸建模。核心洞察是:仅在合成数据上训练的隐式先验可泛化至极具挑战的真实身份与表情,并渲染出细粒度特征(如皱纹、睫毛)。我们利用3D Morphable Face Model生成大规模训练集,对每种身份渲染不同表情、发型、服饰等。随后在该合成数据上训练条件神经辐射场先验,在推理时仅用极少量真实图像对单个主体进行微调。平均仅需三张输入即可跨越合成到真实域的鸿沟。最终个性化3D模型能重建强烈个性化的面部表情,在感知与像素级质量上均优于当前最优方法。

原文摘要 · Abstract (English)

Volumetric modeling and neural radiance field representations have revolutionized 3D face capture and photorealistic novel view synthesis. However, these methods often require hundreds of multi-view input images and are thus inapplicable to cases with less than a handful of inputs. We present a novel volumetric prior on human faces that allows for high-fidelity expressive face modeling from as few as three input views captured in the wild. Our key insight is that an implicit prior trained on synthetic data alone can generalize to extremely challenging real-world identities and expressions and render novel views with fine idiosyncratic details like wrinkles and eyelashes. We leverage a 3D Morphable Face Model to synthesize a large training set, rendering each identity with different expressions, hair, clothing, and other assets. We then train a conditional Neural Radiance Field prior on this synthetic dataset and, at inference time, fine-tune the model on a very sparse set of real images of a single subject. On average, the fine-tuning requires only three inputs to cross the synthetic-to-real domain gap. The resulting personalized 3D model reconstructs strong idiosyncratic facial expressions and outperforms the state-of-the-art in high-quality novel view synthesis of faces from sparse inputs in terms of perceptual and photo-metric quality.

3D人脸新视角生成少样本学习神经辐射场

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。