无需训练,通过视觉交互实现语言引导分割的精准推理
Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation

- 构建视觉-语言交互循环,让模型在图像上直接推理空间关系
- 在多种任务上达到顶尖训练方法的水平,且不需参数更新
- 适合需要快速部署、无需微调的视觉理解场景
语言引导分割突破了传统语义分割的类别限制,可根据自然语言指令分割任意目标区域。现有方法通常采用两阶段框架:先用多模态大语言模型(MLLM)解析指令生成视觉提示,再由基础分割模型(如SAM)生成掩码。然而,由于现成MLLM的空间定位能力有限,这些方法往往依赖大规模数据集上的大量训练才能获得良好精度。尽管近期研究引入了推理机制提升性能,但大多仅在文本域进行链式思考,缺乏直接视觉反馈。本文提出Seg-Agent,一个完全无需训练的框架,首次实现显式的多模态链式推理。不同于仅基于文本的推理,我们的方法构建包含生成、选择、精炼三阶段的交互式视觉推理循环,利用集合标记(SoM)视觉提示将候选区域直接渲染到图像上,使MLLM能够“看见”并迭代推理空间关系。这一显式多模态交互使Seg-Agent在不更新参数的情况下达到与先进训练方法相当的性能。此外,为全面评估跨场景泛化能力,我们引入新基准Various-LangSeg,涵盖显式语义、通用物体和推理引导分割任务。大量实验验证了方法的有效性与鲁棒性。
原文摘要 · Abstract (English)
Language-guided segmentation transcends the scope limitations of traditional semantic segmentation, enabling models to segment arbitrary target regions based on natural language instructions. Existing approaches typically adopt a two-stage framework: employing Multimodal Large Language Models (MLLMs) to interpret instructions and generate visual prompts, followed by foundational segmentation models (e.g., SAM) to produce masks. However, due to the limited spatial grounding capabilities of off-the-shelf MLLMs, these methods often rely on extensive training on large-scale datasets to achieve satisfactory accuracy. While recent advances have introduced reasoning mechanisms to improve performance, they predominantly operate within the textual domain, performing chain-of-thought reasoning solely based on abstract text representations without direct visual feedback. In this paper, we propose Seg-Agent, a completely training-free framework that pioneers Explicit Multimodal Chain-of-Reasoning. Unlike prior text-only reasoning, our approach constructs an interactive visual reasoning loop comprising three stages: generation, selection, and refinement. Specifically, we leverage Set-of-Mark (SoM) visual prompting to render candidate regions directly onto the image, allowing the MLLM to ``see'' and iteratively reason about spatial relationships in the visual domain rather than just the textual one. This explicit multimodal interaction enables Seg-Agent to achieve performance comparable to state-of-the-art training-based methods without any parameter updates. Furthermore, to comprehensively evaluate generalization across diverse scenarios, we introduce Various-LangSeg, a novel benchmark covering explicit semantic, generic object, and reasoning-guided segmentation tasks. Extensive experiments demonstrate the effectiveness and robustness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。