让AI像人一样分步思考并自我修正,提升复杂图像分割的准确性。
CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction
- 利用大模型分步解析问题,从图像中提取细粒度语义信息。
- 通过自检自纠机制,显著降低模糊或复杂场景下的分割错误。
- 无需训练,可外接知识库,适合处理长尾和开放域任务。
现有推理分割方法在复杂查询和域外图像上表现不佳。受链式思维启发,本文提出CoT-Seg——一种无需训练的框架,将链式思维与自校正结合。该框架利用预训练多模态大模型(GPT-4o)的能力,将复杂查询分解为元指令,从图像中提取细粒度语义,并在隐含或复杂提示下定位目标对象。此外,引入自校正阶段:模型评估自身分割结果与原始查询及推理过程的一致性,识别偏差并迭代优化掩码。该机制显著提升在模糊或易错场景下的可靠性。同时支持检索增强推理,可在信息不足时调用外部知识。为验证其能力,我们构建新数据集ReasonSeg-Hard。结果表明,链式思维与自校正相结合,为视觉语言融合分割提供了强大范式。
原文摘要 · Abstract (English)
Existing works of reasoning segmentation often fall short in complex cases, particularly when addressing complicated queries and out-of-domain images. Inspired by the chain-of-thought reasoning, where harder problems require longer thinking steps/time, this paper aims to explore a system that can think step-by-step, look up information if needed, generate results, self-evaluate its own results, and refine the results, in the same way humans approach harder questions. We introduce CoT-Seg, a training-free framework that rethinks reasoning segmentation by combining chain-of-thought reasoning with self-correction. Instead of fine-tuning, CoT-Seg leverages the inherent reasoning ability of pre-trained MLLMs (GPT-4o) to decompose queries into meta-instructions, extract fine-grained semantics from images, and identify target objects even under implicit or complex prompts. Moreover, CoT-Seg incorporates a self-correction stage: the model evaluates its own segmentation against the original query and reasoning trace, identifies mismatches, and iteratively refines the mask. This tight integration of reasoning and correction significantly improves reliability and robustness, especially in ambiguous or error-prone cases. Furthermore, our CoT-Seg framework allows easy incorporation of retrieval-augmented reasoning, enabling the system to access external knowledge when the input lacks sufficient information. To showcase CoT-Seg's ability to handle very challenging cases ,we introduce a new dataset ReasonSeg-Hard. Our results highlight that combining chain-of-thought reasoning, self-correction, offers a powerful paradigm for vision-language integration driven segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。