用AI还原被头显遮挡的面部,让虚拟现实交流更自然
Eye-See-You: Reverse Pass-Through VR and Head Avatars
- 通过生成模型从部分面部信息重建完整人脸图像
- 构建20万样本的VR-Face数据集,覆盖多种遮挡与光照条件
- 适合做虚拟会议、社交互动的沉浸式体验研究者
虚拟现实(VR)头显虽是数字生态的关键组成部分,但其遮挡用户眼睛和部分面部的问题,严重阻碍视觉交流,可能加剧社交孤立。为解决此问题,我们提出RevAvatar框架,利用先进生成模型与多模态AI技术,实现反向透传功能,从部分可见的眼部及下脸区域重建高保真2D面部图像,并生成精准3D头像。该框架推动了AI for Tech的进展,促进虚拟与物理环境的无缝交互,适用于虚拟会议与社交场景。此外,我们构建了包含20万样本的VR-Face数据集,模拟多种VR特有条件,如遮挡、光照变化和畸变。通过克服现有VR系统的核心局限,RevAvatar展示了AI与下一代技术融合的变革潜力,为提升虚拟环境中的人际连接提供了坚实平台。
原文摘要 · Abstract (English)
Virtual Reality (VR) headsets, while integral to the evolving digital ecosystem, present a critical challenge: the occlusion of users' eyes and portions of their faces, which hinders visual communication and may contribute to social isolation. To address this, we introduce RevAvatar, an innovative framework that leverages AI methodologies to enable reverse pass-through technology, fundamentally transforming VR headset design and interaction paradigms. RevAvatar integrates state-of-the-art generative models and multimodal AI techniques to reconstruct high-fidelity 2D facial images and generate accurate 3D head avatars from partially observed eye and lower-face regions. This framework represents a significant advancement in AI4Tech by enabling seamless interaction between virtual and physical environments, fostering immersive experiences such as VR meetings and social engagements. Additionally, we present VR-Face, a novel dataset comprising 200,000 samples designed to emulate diverse VR-specific conditions, including occlusions, lighting variations, and distortions. By addressing fundamental limitations in current VR systems, RevAvatar exemplifies the transformative synergy between AI and next-generation technologies, offering a robust platform for enhancing human connection and interaction in virtual environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。