从被遮挡的YouTube视频重建可动画3D人物,突破传统方法依赖全可见输入的限制。
AHOY! Animatable Humans under Occlusion from YouTube Videos with Gaussian Splatting and Video Diffusion Priors

- 用扩散模型生成缺失身体区域的监督信号,实现遮挡下的完整重建。
- 两阶段结构从稀疏观测逐步构建姿态相关高斯贴图,提升重建精度。
- 头部与身体分开监督,有效保留面部特征,适合真实场景应用。
我们提出AHOY,一种从真实世界单目视频中重建完整、可动画3D高斯角色的方法,即使存在严重遮挡。现有方法通常假设输入为无遮挡且姿态标准的主体,排除了绝大多数包含家具、物体或他人遮挡的真实视频。此类视频重建面临根本挑战:大量身体区域从未被观察到,且无法获得每姿态的多视角监督。我们通过四项贡献解决此问题:(i) 利用身份微调的扩散模型生成此前未观测区域的密集监督信号;(ii) 采用两阶段从规范姿态到姿态依赖的架构,从稀疏观测逐步构建完整高斯贴图;(iii) 分离地图姿态与绑定姿态,吸收生成数据中的多视图不一致;(iv) 采用头/身分离监督策略,保持面部身份一致性。我们在YouTube视频及具有显著遮挡的多视角采集数据上评估,结果达到当前最优重建质量。同时证明生成角色可成功用于新姿态动画,并与手机拍摄的3DGS场景融合。项目页面见https://miraymen.github.io/ahoy/
原文摘要 · Abstract (English)
We present AHOY, a method for reconstructing complete, animatable 3D Gaussian avatars from in-the-wild monocular video despite heavy occlusion. Existing methods assume unoccluded input-a fully visible subject, often in a canonical pose-excluding the vast majority of real-world footage where people are routinely occluded by furniture, objects, or other people. Reconstructing from such footage poses fundamental challenges: large body regions may never be observed, and multi-view supervision per pose is unavailable. We address these challenges with four contributions: (i) a hallucination-as-supervision pipeline that uses identity-finetuned diffusion models to generate dense supervision for previously unobserved body regions; (ii) a two-stage canonical-to-pose-dependent architecture that bootstraps from sparse observations to full pose-dependent Gaussian maps; (iii) a map-pose/LBS-pose decoupling that absorbs multi-view inconsistencies from the generated data; (iv) a head/body split supervision strategy that preserves facial identity. We evaluate on YouTube videos and on multi-view capture data with significant occlusion and demonstrate state-of-the-art reconstruction quality. We also demonstrate that the resulting avatars are robust enough to be animated with novel poses and composited into 3DGS scenes captured using cell-phone video. Our project page is available at https://miraymen.github.io/ahoy/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。