arXiv:2412.14233cs.CV2024-12被引 8

用视觉属性增强图像描述,让模型看得更细更准。

Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception

  • 引入预训练视觉专长模型提取物体细节与关系
  • 融合深度、情绪、交互等属性提升描述精度
  • 适合需要精细视觉理解的多模态任务

大型多模态模型(LMMs)的训练依赖于连接图像与语言的描述性图像标题。现有生成方法常通过蒸馏预训练LMM的标题、从公开网络图像构建或人工标注生成,但在复杂视觉推理任务中,其精确度和粒度往往不足。本文提出EDC方法,利用未专门用于图像标题的现成视觉专长模型,挖掘物体的低级与细粒度属性(如深度、情绪、细粒度类别)及物体间关系(如相对位置、人-物交互)。通过系统整合这些丰富属性,生成更细致的描述性标题,显著提升标题的描述质量,实现对视觉内容更深入、更细腻的理解。实验表明,该方法在视觉理解与依赖精准视觉理解的推理任务中均有效提升性能。EDC全流程代码与数据集将开源至https://github.com/syp2ysy/DCE。

原文摘要 · Abstract (English)

Training Large Multimodality Models (LMMs) relies on descriptive image caption that connects image and language. Existing methods for generating such captions often rely on distilling the captions from pretrained LMMs, constructing them from publicly available internet images, or even generating them through human annotation. However, these strategies can fall short in terms of precision and granularity, particularly when dealing with complex visual reasoning tasks. In this paper, we propose to leverage off-the-shelf visual specialists, which were trained from annotated images initially not for image captioning, for enhancing the image caption. Our approach, named EDC, explores object low-level and fine-grained attributes (e.g., depth, emotion and fine-grained categories) and object relations (e.g., relative location and human-object-interaction (HOI)), and combine the attributes into the descriptive caption. By systematically integrating these rich attributes into the generated captions, EDC significantly improves the descriptive quality of the captions, providing a deeper and more nuanced understanding of the visual content. Experiments demonstrate that such visual specialists are able to improve the performance for visual understanding tasks as well as reasoning that benefits from more accurate visual understanding. The complete source code of EDC pipeline and datasets will be available at https://github.com/syp2ysy/DCE.

多模态图像描述视觉属性细粒度理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。