用3D几何信息生成多人身体细节,解决遮挡与身份一致难题
PersonaCraft: Personalized and Controllable Full-Body Multi-Human Scene Generation Using Occlusion-Aware 3D-Conditioned Diffusion
- 结合SMPLx和扩散模型,用3D深度/法线图实现精准姿态控制
- 在遮挡区域提升细节还原,相比基线方法图像质量提升27.3%
- 支持全身个性化生成,适合虚拟形象、影视制作等场景
我们提出PersonaCraft,一种可控且对遮挡鲁棒的复杂场景中多个人体全身影像合成框架。现有方法在遮挡密集场景中难以实现完整身体个性化,因2D姿态条件缺乏3D几何信息,常导致遮挡模糊与解剖畸变,且多数仅关注面部身份。相比之下,PersonaCraft将扩散模型与3D人体建模结合,采用SMPLx-ControlNet,利用深度图和法线图实现3D感知的姿态条件化,增强解剖一致性。为处理细粒度遮挡,提出遮挡边界增强网络,通过深度边缘信号与遮挡聚焦训练,并设计遮挡感知无分类器引导策略,在遮挡区域选择性强化条件控制而不影响未遮挡区域。PersonaCraft可与面部身份控制网组合,实现全身多人体个性化,显著超越以往仅关注面部身份的方法。其双路径体态表示(基于SMPLx的体形参数与文本精修)支持精确全身个性化与灵活体态调整。大量定量实验与用户研究显示,PersonaCraft在生成高质量、多人体图像方面显著优于现有方法,具备准确身份保持与强遮挡处理能力。
原文摘要 · Abstract (English)
We present PersonaCraft, a framework for controllable and occlusion-robust full-body personalized image synthesis of multiple individuals in complex scenes. Current methods struggle with occlusion-heavy scenarios and complete body personalization, as 2D pose conditioning lacks 3D geometry, often leading to ambiguous occlusions and anatomical distortions, and many approaches focus solely on facial identity. In contrast, our PersonaCraft integrates diffusion models with 3D human modeling, employing SMPLx-ControlNet, to utilize 3D geometry like depth and normal maps for robust 3D-aware pose conditioning and enhanced anatomical coherence. To handle fine-grained occlusions, we propose Occlusion Boundary Enhancer Network that exploits depth edge signals with occlusion-focused training, and Occlusion-Aware Classifier-Free Guidance strategy that selectively reinforces conditioning in occluded regions without affecting unoccluded areas. PersonaCraft can seamlessly be combined with Face Identity ControlNet, achieving full-body multi-human personalization and thus marking a significant advancement beyond prior approaches that concentrate only on facial identity. Our dual-pathway body shape representation with SMPLx-based shape parameters and textual refinement, enables precise full-body personalization and flexible user-defined body shape adjustments. Extensive quantitative experiments and user studies demonstrate that PersonaCraft significantly outperforms existing methods in generating high-quality, multi-person images with accurate personalization and robust occlusion handling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。