arXiv:2409.06481cs.CV2024-09CVPR被引 3

构建首个大规模否定指令图像编辑数据集,解决模型理解‘不要’指令难题。

NeIn: Telling What You Don't Want

  • 基于MS-COCO自动构建含否定语句的图像编辑数据集
  • 36万条样本验证显示顶尖模型仍难理解否定指令
  • 适合研究视觉语言模型在复杂语义理解上的短板

否定是人类表达不想要内容的基本语言机制,但现有文本引导图像编辑研究对此关注极少。这导致视觉语言模型(VLM)难以准确理解否定指令,进而影响编辑效果。当前缺乏标准化评估数据集,制约了该方向研究进展。本文提出首个大规模否定指令数据集NeIn,包含366,957个五元组(源图像、原始描述、目标物体、否定句、目标图像),其中342,775条用于训练,24,182条用于基准测试。数据通过两步生成:首先利用BLIP与InstructPix2Pix(微调于MagicBrush数据集)生成样本和否定短语;随后使用BLIP与LLaVA-NeXT进行错误样本过滤。同时提出评估协议,对多种VLM在文本引导图像编辑任务中进行实验,结果表明即使最新SOTA模型也难以有效理解否定查询。

原文摘要 · Abstract (English)

Negation is a fundamental linguistic concept used by humans to convey information that they do not desire. Despite this, minimal research has focused on negation within text-guided image editing. This lack of research means that vision-language models (VLMs) for image editing may struggle to understand negation, implying that they struggle to provide accurate results. One barrier to achieving human-level intelligence is the lack of a standard collection by which research into negation can be evaluated. This paper presents the first large-scale dataset, Negative Instruction (NeIn), for studying negation within instruction-based image editing. Our dataset comprises 366,957 quintuplets, i.e., source image, original caption, selected object, negative sentence, and target image in total, including 342,775 queries for training and 24,182 queries for benchmarking image editing methods. Specifically, we automatically generate NeIn based on a large, existing vision-language dataset, MS-COCO, via two steps: generation and filtering. During the generation phase, we leverage two VLMs, BLIP and InstructPix2Pix (fine-tuned on MagicBrush dataset), to generate NeIn's samples and the negative clauses that expresses the content of the source image. In the subsequent filtering phase, we apply BLIP and LLaVA-NeXT to remove erroneous samples. Additionally, we introduce an evaluation protocol to assess the negation understanding for image editing models. Extensive experiments using our dataset across multiple VLMs for text-guided image editing demonstrate that even recent state-of-the-art VLMs struggle to understand negative queries.

图像编辑否定理解视觉语言模型数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。