用偏好优化提升单图人体网格恢复的对齐与鲁棒性
ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization
- 通过评分模型生成偏好数据，直接优化扩散模型的人体网格对齐
- 在无3D标注的野外图像上仍能有效评估并提升模型性能
- 适用于真实场景下人体重建，尤其适合数据有限的场景
从单张图像进行人体网格恢复（HMR）因深度模糊和遮挡问题本质为病态问题。概率方法虽可生成多个合理预测，但常与2D图像观测不一致，且对野外图像鲁棒性差。为此，我们提出ADHMR框架，通过偏好优化对齐基于扩散模型的HMR。首先训练一个无需3D标注即可评估任意图像中人体网格预测的评分模型HMR-Scorer；随后利用该模型构建偏好数据集，每张图像包含一对优劣预测结果，并以此对基础模型进行直接偏好优化微调。此外，HMR-Scorer还能通过数据清洗提升现有HMR模型性能，即使训练样本较少亦可。大量实验表明，ADHMR优于当前最先进方法。代码已公开：https://github.com/shenwenhao01/ADHMR。
原文摘要 · Abstract (English)
Human mesh recovery (HMR) from a single image is inherently ill-posed due to depth ambiguity and occlusions. Probabilistic methods have tried to solve this by generating numerous plausible 3D human mesh predictions, but they often exhibit misalignment with 2D image observations and weak robustness to in-the-wild images. To address these issues, we propose ADHMR, a framework that Aligns a Diffusion-based HMR model in a preference optimization manner. First, we train a human mesh prediction assessment model, HMR-Scorer, capable of evaluating predictions even for in-the-wild images without 3D annotations. We then use HMR-Scorer to create a preference dataset, where each input image has a pair of winner and loser mesh predictions. This dataset is used to finetune the base model using direct preference optimization. Moreover, HMR-Scorer also helps improve existing HMR models by data cleaning, even with fewer training samples. Extensive experiments show that ADHMR outperforms current state-of-the-art methods. Code is available at: https://github.com/shenwenhao01/ADHMR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。