arXiv:2602.03595cs.CV2026-02被引 5

用多智能体协作与反思机制,提升视频目标分割的准确性与灵活性。

Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation

  • 分步推理+动态聚焦,提升帧选择与文本相关性。
  • 自反思链使系统可验证中间结果并优化后续推理。
  • 零样本部署快,支持新模型无缝接入,无需微调。

指代视频目标分割(RVOS)旨在根据文本查询分割视频中的对象。现有方法主要依赖多模态大语言模型(MLLM)的大规模监督微调(SFT),但该范式数据依赖重、难以随MLLM快速演进。尽管近期零样本方法更具灵活性,但因流程设计简单,性能仍显著落后于SFT方法。为此,我们提出 extbf{Refer-Agent},一个具有交替推理-反思机制的协作多智能体系统。该系统将RVOS分解为逐步推理过程:推理阶段引入从粗到精的帧选择策略,确保帧多样性与文本相关性,并采用动态聚焦布局自适应调整视觉关注区域;反思阶段提出链式自反思机制,通过提问-回应对生成反思链,验证中间结果并反馈用于下一轮推理优化。在五个挑战性基准上的大量实验表明,Refer-Agent显著优于当前最优方法,涵盖SFT模型与零样本方案。此外,Refer-Agent具备高度灵活性,可无额外微调成本快速集成新MLLM。代码将发布于https://github.com/iSEE-Laboratory/Refer-Agent。

原文摘要 · Abstract (English)

Referring Video Object Segmentation (RVOS) aims to segment objects in videos based on textual queries. Current methods mainly rely on large-scale supervised fine-tuning (SFT) of Multi-modal Large Language Models (MLLMs). However, this paradigm suffers from heavy data dependence and limited scalability against the rapid evolution of MLLMs. Although recent zero-shot approaches offer a flexible alternative, their performance remains significantly behind SFT-based methods, due to the straightforward workflow designs. To address these limitations, we propose \textbf{Refer-Agent}, a collaborative multi-agent system with alternating reasoning-reflection mechanisms. This system decomposes RVOS into step-by-step reasoning process. During reasoning, we introduce a Coarse-to-Fine frame selection strategy to ensure the frame diversity and textual relevance, along with a Dynamic Focus Layout that adaptively adjusts the agent's visual focus. Furthermore, we propose a Chain-of-Reflection mechanism, which employs a Questioner-Responder pair to generate a self-reflection chain, enabling the system to verify intermediate results and generates feedback for next-round reasoning refinement. Extensive experiments on five challenging benchmarks demonstrate that Refer-Agent significantly outperforms state-of-the-art methods, including both SFT-based models and zero-shot approaches. Moreover, Refer-Agent is flexible and enables fast integration of new MLLMs without any additional fine-tuning costs. Code will be released at https://github.com/iSEE-Laboratory/Refer-Agent.

视频分割多智能体自反思零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。