利用扩散模型隐空间特征,提升人体姿态估计的跨域鲁棒性。
SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation
- 从扩散模型U-Net不同层级提取特征,选择最优层用于姿态估计。
- 在COCO-OOD上超越现有方法,尤其在风格迁移和噪声干扰下表现优异。
- 适合需要强泛化能力的开放域姿态估计场景,如真实世界应用。
预训练扩散模型在U-Net各层级提供丰富的潜在特征,正成为强大的视觉骨干网络。尽管先前工作如Marigold和Lotus将其用于深度图、法向量等密集几何感知任务,其在跨域人体姿态估计中的潜力仍待探索。通过系统分析Stable Diffusion U-Net不同上采样层级的潜在特征,我们识别出对姿态估计最具鲁棒性和跨域泛化能力的层级。基于此,我们提出SDPose:(i) 从选定上采样块中提取U-Net特征;(ii) 使用轻量级特征融合模块构建鲁棒表征;(iii) 联合优化关键点热图监督与辅助潜在空间重建损失,以正则化训练并保留预训练生成先验。为评估跨域泛化与鲁棒性,我们构建了基于COCO的COCO-OOD基准,包含三个风格迁移子集和一个含噪声、天气、数字伪影与模糊的损坏子集。在更短微调周期下,SDPose在COCO上性能接近Sapiens,超越Sapiens-1B在COCO-WholeBody上的表现,并在HumanArt和COCO-OOD上建立新SOTA。
原文摘要 · Abstract (English)
Pre-trained diffusion models provide rich latent features across U-Net levels and are emerging as powerful vision backbones. While prior works such as Marigold and Lotus repurpose diffusion priors for dense geometric perception tasks such as depth and surface normal estimation, their potential for cross-domain human pose estimation remains largely unexplored. Through a systematic analysis of latent features from different upsampling levels of the Stable Diffusion U-Net, we identify the levels that deliver the strongest robustness and cross-domain generalization for pose estimation. Building on these findings, we propose \textbf{SDPose}, which (i) extracts U-Net features from the selected upsampling blocks, (ii) fuses them with a lightweight feature aggregation module to form a robust representation, and (iii) jointly optimizes keypoint heatmap supervision with an auxiliary latent reconstruction loss to regularize training and preserve the pre-trained generative prior. To evaluate cross-domain generalization and robustness, we construct COCO-OOD, a COCO-based benchmark with four subsets: three style-transferred splits to assess domain shift, and one corruption split (noise, weather, digital artifacts, and blur) to test robustness. With a shorter fine-tuning schedule, SDPose achieves performance comparable to Sapiens on COCO, surpasses Sapiens-1B on COCO-WholeBody, and establishes new state-of-the-art results on HumanArt and COCO-OOD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。