arXiv:2503.07503cs.CV2025-03被引 7

用GPT的思维链提升复杂图像分割质量,无需训练即可用文本和涂鸦精准定位目标。

Think Before You Segment: High-Quality Reasoning Segmentation with GPT Chain of Thoughts

  • 先让GPT生成图像的详细思维链描述,再指导分割模型
  • 零样本下显著提升复杂场景分割效果,对提示词不敏感
  • 支持文本与手绘涂鸦交互,适合非专业用户精细调整

推理分割是一项具有挑战性的视觉-语言任务,旨在根据复杂、隐含甚至非视觉的文本查询输出对应的分割掩码。以往方法将多模态大语言模型(MLLM)与分割模型结合,但在处理域外物体、结构复杂、边界模糊、遮挡或与周围高度相似的情况时,分割质量仍不理想。本文提出ThinkFirst,一种无需训练的推理分割框架,利用GPT的思维链(Chain of Thought)能力应对这些挑战。该方法让GPT-4o等强大多模态模型生成对图像的详细思维链描述,再由语言引导的分割助手依据该描述进行分割。用户可通过文本或图像涂鸦与分割代理交互,实现逐步优化或沟通。在多种对象上的实验表明,此零样本思维链方法显著优于基线模型,在定性和定量上均有提升,且对用户输入提示的敏感性大幅降低。

原文摘要 · Abstract (English)

Reasoning segmentation is a challenging vision-language task that aims to output the segmentation mask with respect to a complex, implicit, and even non-visual query text. Previous works incorporated multimodal Large Language Models (MLLMs) with segmentation models to approach the difficult problem. However, their segmentation quality often falls short in complex cases, particularly when dealing with out-of-domain objects with intricate structures, blurry boundaries, occlusions, or high similarity with surroundings. In this paper, we introduce ThinkFirst, a training-free reasoning segmentation framework that leverages GPT's chain of thought to address these challenging cases. Our approach allows GPT-4o or other powerful MLLMs to generate a detailed, chain-of-thought description of an image. This summarized description is then passed to a language-instructed segmentation assistant to aid the segmentation process. Our framework allows users to easily interact with the segmentation agent using multimodal inputs, such as easy text and image scribbles, for successive refinement or communication. We evaluate the performance of ThinkFirst on diverse objects. Extensive experiments show that, this zero-shot-CoT approach significantly improves the vanilla reasoning segmentation agent, both qualitatively and quantitatively, while being less sensitive or critical to user-supplied prompts after Thinking First.

图像分割思维链多模态零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。