让大模型先看物体轨迹再推理，提升零样本视频目标分割精度
AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation
- 先用SAM3生成全时序物体掩码轨迹，提供可靠感知基础
- 大模型基于物体轨迹进行查询引导的迭代推理，准确率超越现有方法
- 无需训练，适配多种大模型，适用于零样本视频分割任务
指代视频目标分割（RVOS）旨在根据自然语言查询在视频中分割目标对象。训练无关的方法通常遵循固定流程：多模态大模型（MLLM）选择关键帧，在这些帧中定位被提及对象，再由视频分割模型传播结果。然而，该设计要求MLLM在缺乏物体层面证据前做出时间决策，限制了推理质量和时空覆盖范围。为此，我们提出AgentRVOS，一种基于SAM3与MLLM互补优势的训练无关智能体框架。给定查询中的概念，SAM3通过生成掩码轨迹，在全时空范围内提供可靠感知。随后，MLLM在这一物体级证据上进行查询引导的推理，通过SAM3的时间存在信息迭代修剪。大量实验表明，AgentRVOS在多个基准上均达到训练无关方法的最先进性能，且对多种MLLM骨干网络保持一致表现。
原文摘要 · Abstract (English)
Referring Video Object Segmentation (RVOS) aims to segment a target object throughout a video given a natural language query. Training-free methods for this task follow a common pipeline: a MLLM selects keyframes, grounds the referred object within those frames, and a video segmentation model propagates the results. While intuitive, this design asks the MLLM to make temporal decisions before any object-level evidence is available, limiting both reasoning quality and spatio-temporal coverage. To overcome this, we propose AgentRVOS, a training-free agentic pipeline built on the complementary strengths of SAM3 and a MLLM. Given a concept derived from the query, SAM3 provides reliable perception over the full spatio-temporal extent through generated mask tracks. The MLLM then identifies the target through query-grounded reasoning over this object-level evidence, iteratively pruning guided by SAM3's temporal existence information. Extensive experiments show that AgentRVOS achieves state-of-the-art performance among training-free methods across multiple benchmarks, with consistent results across diverse MLLM backbones. Our project page is available at: https://cvlab-kaist.github.io/AgentRVOS/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。