用单视角视频修复头戴设备遮挡的面部,同时重建3D人脸。
Geometry-Aware Video Inpainting for Joint Headset Occlusion Removal and Face Reconstruction in Social XR
- 基于生成对抗网络和关键点引导,修复遮挡区域。
- 可恢复真实感3D人脸模型,保持身份一致性。
- 适用于社交元宇宙中的实时表情与视线交互。
头戴式显示设备(HMD)是体验扩展现实(XR)环境的关键,但会遮挡用户面部上半部分,影响外部视频录制,严重削弱远程会议等社交XR应用中面部表情与眼神交流的沉浸感。本文提出一种几何感知的学习框架,仅通过单视角RGB视频,联合实现HMD遮挡移除与完整3D面部几何重建。方法采用基于GAN的视频修复网络,结合密集面部关键点与无遮挡参考帧,恢复缺失面部区域并保留身份特征;随后通过SynergyNet模块从修复后的帧回归3D可变形模型(3DMM)参数,实现高精度3D人脸重建。整个流程贯穿密集关键点优化,提升修复质量与几何保真度。实验表明,该框架能有效移除HMD遮挡,生成逼真的3D人脸输出。消融实验显示,即使在关键点稀疏配置下,系统仍保持鲁棒性,仅出现轻微质量下降。
原文摘要 · Abstract (English)
Head-mounted displays (HMDs) are essential for experiencing extended reality (XR) environments and observing virtual content. However, they obscure the upper part of the user's face, complicating external video recording and significantly impacting social XR applications such as teleconferencing, where facial expressions and eye gaze details are crucial for creating an immersive experience. This study introduces a geometry-aware learning-based framework to jointly remove HMD occlusions and reconstruct complete 3D facial geometry from RGB frames captured from a single viewpoint. The method integrates a GAN-based video inpainting network, guided by dense facial landmarks and a single occlusion-free reference frame, to restore missing facial regions while preserving identity. Subsequently, a SynergyNet-based module regresses 3D Morphable Model (3DMM) parameters from the inpainted frames, enabling accurate 3D face reconstruction. Dense landmark optimization is incorporated throughout the pipeline to improve both the inpainting quality and the fidelity of the recovered geometry. Experimental results demonstrate that the proposed framework can successfully remove HMDs from RGB facial videos while maintaining facial identity and realism, producing photorealistic 3D face geometry outputs. Ablation studies further show that the framework remains robust across different landmark densities, with only minor quality degradation under sparse landmark configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。