让AI通过多轮思考与工具调用，逐步修正目标位置，实现更准的文本引导分割。
RSAgent: Learning to Reason and Act for Text-Guided Segmentation via Multi-Turn Tool Invocations
- 采用多轮推理+工具调用机制，动态修正分割位置
- 零样本下在ReasonSeg上达66.5% gIoU，比基线高9%
- 适合需要精细定位与可解释性的视觉理解任务
文本引导目标分割需兼具跨模态推理与像素定位能力。现有方法多将其视为单次定位，模型一次性预测像素提示并驱动外部分割器，难以验证、重聚焦或优化初始错误定位。为此，我们提出RSAgent，一种基于多轮工具调用的智能体式多模态大模型（MLLM），通过交替推理与动作实现迭代分割。RSAgent调用分割工具箱，观察视觉反馈，并结合历史观察修正空间假设，重新定位目标并逐步优化掩码。我们构建数据流水线合成多轮推理分割轨迹，采用两阶段训练框架：先冷启动监督微调，再通过细粒度、任务专属奖励进行智能体强化学习。大量实验表明，RSAgent在ReasonSeg测试集上零样本性能达66.5% gIoU，较Seg-Zero-7B提升9%，在RefCOCOg上达到81.5% cIoU，显著优于当前主流方法，在域内与域外基准均表现领先。
原文摘要 · Abstract (English)
Text-guided object segmentation requires both cross-modal reasoning and pixel grounding abilities. Most recent methods treat text-guided segmentation as one-shot grounding, where the model predicts pixel prompts in a single forward pass to drive an external segmentor, which limits verification, refocusing and refinement when initial localization is wrong. To address this limitation, we propose RSAgent, an agentic Multimodal Large Language Model (MLLM) which interleaves reasoning and action for segmentation via multi-turn tool invocations. RSAgent queries a segmentation toolbox, observes visual feedback, and revises its spatial hypothesis using historical observations to re-localize targets and iteratively refine masks. We further build a data pipeline to synthesize multi-turn reasoning segmentation trajectories, and train RSAgent with a two-stage framework: cold-start supervised fine-tuning followed by agentic reinforcement learning with fine-grained, task-specific rewards. Extensive experiments show that RSAgent achieves a zero-shot performance of 66.5% gIoU on ReasonSeg test, improving over Seg-Zero-7B by 9%, and reaches 81.5% cIoU on RefCOCOg, demonstrating state-of-the-art performance on both in-domain and out-of-domain benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。