用双编码器提升单图3D人脸重建精度，支持多角度真实渲染。
Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images
- 设计双编码器系统，分别优化不同视角下的重建质量。
- 在PanoHead基础上实现360度视角的高保真3D人脸生成。
- 提出遮挡感知的三平面判别器，提升拼接一致性与视觉真实感。
3D GAN反演旨在将单张图像映射到3D生成对抗网络（GAN）的潜在空间，从而实现3D几何重建。现有方法大多基于EG3D，擅长生成近正面视角，难以全面还原多角度3D场景。本文提出新框架，基于擅长360度视角合成的PanoHead，构建双编码器系统，以实现从不同视角出发的高保真重建与真实生成。同时，在三平面域设计拼接框架，整合两编码器输出。为保证拼接无缝，两个编码器需在任务分工下保持结果一致，为此引入基于新型遮挡感知三平面判别器的对抗损失进行精细化训练。实验表明，本方法在定性和定量上均优于现有方法。
原文摘要 · Abstract (English)
3D GAN inversion aims to project a single image into the latent space of a 3D Generative Adversarial Network (GAN), thereby achieving 3D geometry reconstruction. While there exist encoders that achieve good results in 3D GAN inversion, they are predominantly built on EG3D, which specializes in synthesizing near-frontal views and is limiting in synthesizing comprehensive 3D scenes from diverse viewpoints. In contrast to existing approaches, we propose a novel framework built on PanoHead, which excels in synthesizing images from a 360-degree perspective. To achieve realistic 3D modeling of the input image, we introduce a dual encoder system tailored for high-fidelity reconstruction and realistic generation from different viewpoints. Accompanying this, we propose a stitching framework on the triplane domain to get the best predictions from both. To achieve seamless stitching, both encoders must output consistent results despite being specialized for different tasks. For this reason, we carefully train these encoders using specialized losses, including an adversarial loss based on our novel occlusion-aware triplane discriminator. Experiments reveal that our approach surpasses the existing encoder training methods qualitatively and quantitatively. Please visit the project page: https://berkegokmen1.github.io/dual-enc-3d-gan-inv.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。