arXiv:2508.12663cs.CV2025-08被引 1

用扩散模型分两步重建被遮挡的人体结构与外观,效果优于现有方法。

Stable Diffusion-Based Approach for Human De-Occlusion

  • 先用扩散模型生成人体结构掩码,再用稳定扩散恢复颜色图像。
  • 在严重遮挡下仍能准确还原人体外观,显著提升姿态与三维重建性能。
  • 结合视觉问答和CLIP特征,减少生成图像的像素退化问题。

人类能借助先验知识和可见线索推断被遮挡物体的缺失部分,但让深度学习模型准确预测遮挡区域仍具挑战。去遮挡任务旨在重建掩码和RGB外观。本文聚焦人体去遮挡,目标是恢复被遮挡的身体结构与外观。方法分为两个阶段:第一阶段利用基于扩散的人体先验生成完整身体结构表示,并结合遮挡关节热图提供明确的空间线索;第二阶段以重建的无遮挡掩码为条件输入,指导模型对需修复区域进行颜色重建。为增强颜色生成,引入通过视觉问答模型提取并经CLIP编码的人体特异性文本特征。使用稳定扩散进行颜色重建,并微调解码器以缓解因潜在空间变换导致的可见区域像素退化问题。所提方法在严重遮挡下仍能有效重建人体外观,在掩码与颜色重建上均优于现有方法。生成的去遮挡图像还可提升下游人体任务(如2D姿态估计、3D人体重建)的性能。代码将公开。

原文摘要 · Abstract (English)

Humans can infer the missing parts of an occluded object by leveraging prior knowledge and visible cues. However, enabling deep learning models to accurately predict such occluded regions remains a challenging task. De-occlusion addresses this problem by reconstructing both the mask and RGB appearance. In this work, we focus on human de-occlusion, specifically targeting the recovery of occluded body structures and appearances. Our approach decomposes the task into two stages: mask completion and RGB completion. The first stage leverages a diffusion-based human body prior to provide a comprehensive representation of body structure, combined with occluded joint heatmaps that offer explicit spatial cues about missing regions. The reconstructed amodal mask then serves as a conditioning input for the second stage, guiding the model on which areas require RGB reconstruction. To further enhance RGB generation, we incorporate human-specific textual features derived using a visual question answering (VQA) model and encoded via a CLIP encoder. RGB completion is performed using Stable Diffusion, with decoder fine-tuning applied to mitigate pixel-level degradation in visible regions -- a known limitation of prior diffusion-based de-occlusion methods caused by latent space transformations. Our method effectively reconstructs human appearances even under severe occlusions and consistently outperforms existing methods in both mask and RGB completion. Moreover, the de-occluded images generated by our approach can improve the performance of downstream human-centric tasks, such as 2D pose estimation and 3D human reconstruction. The code will be made publicly available.

图像修复扩散模型人体重建

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。