用零样本思维链解析视频，让模型自动找对目标对象
CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos
- 通过零样本思维链分析时空语义，定位复杂查询中的目标物体
- 在显式和隐式查询上均超越现有方法，尤其在动态变化场景表现优异
- 无需训练，兼容闭源大模型，适合实时视频流处理
推理视频对象分割是一项挑战性任务，旨在根据复杂的隐含文本查询从输入视频生成掩码序列。现有方法虽微调多模态大语言模型（MLLM）以应对该任务，但在面对复杂的时间敏感查询时仍表现不佳，暴露出其在复杂场景中时空整合能力的不足。本文提出 CoT-RVS，一种新颖的零样本框架，利用 MLLM 的零样本思维链（CoT）能力进行时序-语义推理：CoT-RVS 分析当前帧中可能匹配语言查询的可见物体（语义），并为每个物体选择一个在所有帧中可轻松观测的关键帧（时序）。值得注意的是，CoT-RVS 框架无需训练，且兼容闭源 MLLM，可应用于推理视频实例分割。其无训练特性还支持扩展至在线视频流处理，测试时使用 CoT 动态更新关注目标，当更优目标出现并变得可见时即时调整。我们在显式与隐式查询下进行了广泛实验，结果表明，CoT-RVS 在定性和定量上均显著优于先前方法。
原文摘要 · Abstract (English)
Reasoning Video Object Segmentation is a challenging task, aiming at generating a mask sequence from an input video given a complex and implicit text query. While existing works finetune Multimodal Large Language Models (MLLM) for the task, they still fail in video inputs given complex temporally-sensitive queries, indicating their lack of temporal and spatial integration in complex scenarios. In this paper, we propose CoT-RVS, a novel framework employing the zero-shot Chain-of-Thought (CoT) capability of MLLM to address these complex challenges by temporal-semantic reasoning: CoT-RVS analyzes the visible objects within a given frame that possibly match the language query (semantic), and chooses a corresponding keyframe for each object that can be observed effortlessly among all frames (temporal). Notably, the CoT-RVS framework is training-free and compatible with closed-source MLLMs, which can be applied to Reasoning Video Instance Segmentation. Our framework's training-free feature further allows its extension to process online video streams, where the CoT is used at test time to update the object of interest when a better target starts to emerge and becomes visible. We conduct extensive experiments on video object segmentation with explicit and implicit queries. The results show that CoT-RVS significantly outperforms previous works in both cases, qualitatively and quantitatively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。