用单目视频重建动态人体,还能补全看不见的身体部位。
WonderHuman: Hallucinating Unseen Parts in Dynamic 3D Human Reconstruction
- 结合2D生成扩散模型先验,补全单视角视频中缺失的身体部分。
- 在未见视角下实现高保真渲染,关键部位重建精度显著提升。
- 适合需要低成本捕捉高质量人体动画的创作者与研究人员。
本文提出WonderHuman,从单目视频重建动态人体虚拟形象,以实现高保真新视角合成。以往方法通常要求输入视频完整覆盖人体,但在日常使用中,如仅能获取前视单目视频时,难以重建未见身体部位。为此,我们引入基于2D生成扩散模型先验的方法,实现从单目视频中高质量、逼真的动态人体重建,包括准确渲染未见部位。本方法提出双空间优化策略,在规范空间与观测空间同时应用得分蒸馏采样(SDS),确保视觉一致性并提升真实感。此外,设计视角选择策略与姿态特征注入机制,强化SDS预测与观测数据的一致性,保证姿态依赖效果与更高重建保真度。实验表明,该方法在生成逼真图像方面达到当前最优性能,尤其在挑战性的未见部位重建上表现突出。
原文摘要 · Abstract (English)
In this paper, we present WonderHuman to reconstruct dynamic human avatars from a monocular video for high-fidelity novel view synthesis. Previous dynamic human avatar reconstruction methods typically require the input video to have full coverage of the observed human body. However, in daily practice, one typically has access to limited viewpoints, such as monocular front-view videos, making it a cumbersome task for previous methods to reconstruct the unseen parts of the human avatar. To tackle the issue, we present WonderHuman, which leverages 2D generative diffusion model priors to achieve high-quality, photorealistic reconstructions of dynamic human avatars from monocular videos, including accurate rendering of unseen body parts. Our approach introduces a Dual-Space Optimization technique, applying Score Distillation Sampling (SDS) in both canonical and observation spaces to ensure visual consistency and enhance realism in dynamic human reconstruction. Additionally, we present a View Selection strategy and Pose Feature Injection to enforce the consistency between SDS predictions and observed data, ensuring pose-dependent effects and higher fidelity in the reconstructed avatar. In the experiments, our method achieves SOTA performance in producing photorealistic renderings from the given monocular video, particularly for those challenging unseen parts. The project page and source code can be found at https://wyiguanw.github.io/WonderHuman/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。