用视觉理解损失提升单图人脸3D重建与编辑质量
PercHead: Perceptual Head Model for Single-Image 3D Head Reconstruction & Editing
- 基于DINOv2和SAM的感知损失替代传统低层损失
- 在新视角合成中达当前最优，极端视角仍稳定
- 支持文本/图像控制外观的解耦3D编辑，交互直观
我们提出PercHead，一个用于单图3D人脸重建与解耦3D编辑的模型。由于输入存在多种合理解释，这两项任务极具挑战性。核心是基于DINOv2和SAM 2.1的新型感知损失，相比广泛使用的LPIPS、SSIM或L1等低层损失，该损失依赖深层视觉理解，提供更泛化的监督信号。我们证明该损失可直接替换标准损失，并显著提升高频区域视觉质量。模型架构采用视觉变换器（ViTs），实现3D表示与2D输入的解耦。在多视角图像上训练以保证视图一致性，在真实场景图像上训练以增强对新环境的泛化能力。实验表明，该模型在新视角合成上达到当前最优性能，且对极端视角具有优异鲁棒性。进一步通过替换编码器并微调网络，扩展为解耦3D编辑：使用分割图控制几何，文本提示或参考图像指定外观。通过交互式GUI展示了直观强大的3D编辑能力。
原文摘要 · Abstract (English)
We present PercHead, a model for single-image 3D head reconstruction and disentangled 3D editing - two tasks that are inherently challenging due to ambiguity in plausible explanations for the same input. At the heart of our approach lies our novel perceptual loss based on DINOv2 and SAM 2.1. Unlike widely-adopted low-level losses like LPIPS, SSIM or L1, we rely on deep visual understanding of images and the resulting generalized supervision signals. We show that our new loss can be a drop-in replacement for standard losses and used to improve visual quality in high-frequency areas. We base our model architecture on Vision Transformers (ViTs), allowing us to decouple the 3D representation from the 2D input. We train our method on multi-view images for view-consistency and in-the-wild images for strong transferability to new environments. Our model achieves state-of-the-art performance in novel-view synthesis and, furthermore, exhibits exceptional robustness to extreme viewing angles. We also extend our base model to disentangled 3D editing by swapping the encoder and fine-tuning the network. A segmentation map controls geometry and either a text prompt or a reference image specifies appearance. We highlight the intuitive and powerful 3D editing capabilities through an interactive GUI. Project Page: https://antoniooroz.github.io/PercHead Video: https://www.youtube.com/watch?v=4hFybgTk4kE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。