用大模型自动生成人体交互数据，解决亲密接触场景下训练数据少的问题。
Ask, Pose, Unite: Scaling Data Acquisition for Close Interactions with Vision Language Models
- 利用视觉语言模型自动标注接触图，指导测试时优化生成伪真值网格
- 构建含6200+对接触人体网格的APU数据集，覆盖多种互动类型
- 适合研究人体姿态估计、交互建模及扩散模型应用的研究者
亲密人际互动中的社会动态给人体网格估计（HME）带来巨大挑战，主要源于物理接触的复杂性以及训练数据稀缺。为此，我们提出一种新型数据生成方法，利用大视觉语言模型（LVLM）标注接触图，引导测试时优化生成配对的图像与伪真值网格。该方法不仅减轻了人工标注负担，还实现了针对亲密互动场景的全面数据集构建。我们构建的Ask Pose Unite（APU）数据集包含超过6.2k对处于接触状态的人体网格，涵盖多样化互动类型，源自自然人-人场景图像。实验证明，使用该数据集训练基于扩散模型的接触先验，在优化过程中作为引导，显著提升对未见互动场景的网格估计性能。本工作有效缓解了HME中亲密互动数据稀缺的长期难题，增强了模型处理复杂交互场景的能力。
原文摘要 · Abstract (English)
Social dynamics in close human interactions pose significant challenges for Human Mesh Estimation (HME), particularly due to the complexity of physical contacts and the scarcity of training data. Addressing these challenges, we introduce a novel data generation method that utilizes Large Vision Language Models (LVLMs) to annotate contact maps which guide test-time optimization to produce paired image and pseudo-ground truth meshes. This methodology not only alleviates the annotation burden but also enables the assembly of a comprehensive dataset specifically tailored for close interactions in HME. Our Ask Pose Unite (APU) dataset, comprising over 6.2k human mesh pairs in contact covering diverse interaction types, is curated from images depicting naturalistic person-to-person scenes. We empirically show that using our dataset to train a diffusion-based contact prior, used as guidance during optimization, improves mesh estimation on unseen interactions. Our work addresses longstanding challenges of data scarcity for close interactions in HME enhancing the field's capabilities of handling complex interaction scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。