用人体关键点增强视觉语言模型对动作的理解能力。
LLaVA-Pose: Enhancing Human Pose and Action Understanding via Keypoint-Integrated Instruction Tuning
- 将人体关键点与图文描述结合，生成专用指令数据。
- 在20万样本上微调后，性能比原模型提升33.2%。
- 适合需要精细动作理解的科研与应用开发者。
当前视觉语言模型在通用视觉理解任务中表现良好，但在涉及人体姿态与动作的复杂任务中表现不足，原因在于缺乏专门的视觉语言指令遵循数据。本文通过将人体关键点与传统视觉特征（如描述、边界框）融合，构建了一种生成此类数据的方法，以实现对以人为中心场景的更精准理解。我们构造了一个包含200,328个样本的数据集，用于微调模型，重点覆盖对话、详细描述和复杂推理三个领域。为此，我们建立了扩展的人体姿态与动作理解基准（E-HPAUB），用于评估模型性能。基于该数据集微调的LLaVA-Pose模型在基准测试中相比原始的LLaVA-1.5-7B模型整体提升了33.2%。结果表明，关键点融合数据能有效提升多模态模型在以人为中心视觉理解任务中的表现。代码已开源于https://github.com/Ody-trek/LLaVA-Pose。
原文摘要 · Abstract (English)
Current vision-language models (VLMs) are well-adapted for general visual understanding tasks. However, they perform inadequately when handling complex visual tasks related to human poses and actions due to the lack of specialized vision-language instruction-following data. We introduce a method for generating such data by integrating human keypoints with traditional visual features such as captions and bounding boxes, enabling more precise understanding of human-centric scenes. Our approach constructs a dataset comprising 200,328 samples tailored to fine-tune models for human-centric tasks, focusing on three areas: conversation, detailed description, and complex reasoning. We establish an Extended Human Pose and Action Understanding Benchmark (E-HPAUB) to assess model performance on human pose and action understanding. We fine-tune the LLaVA-1.5-7B model using this dataset and evaluate our resulting LLaVA-Pose model on the benchmark, achieving significant improvements. Experimental results show an overall improvement of 33.2% compared to the original LLaVA-1.5-7B model. These findings highlight the effectiveness of keypoint-integrated data in enhancing multimodal models for human-centric visual understanding. Code is available at https://github.com/Ody-trek/LLaVA-Pose.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。