用大模型分步推理,零训练实现精准视频目标分割。
Unleashing Hierarchical Reasoning: An LLM-Driven Framework for Training-Free Referring Video Object Segmentation
- 将语言描述拆解为结构化指令,分步引导视觉定位
- 在三个基准上达到当前最优,最高提升3.2% mIoU
- 适合需要快速部署、无需训练的视频理解场景
指代式视频目标分割(RVOS)旨在根据语言描述在视频中分割出目标对象。核心挑战在于将静态文本与动态视觉内容对齐,尤其当物体外观相似但运动和姿态不一致时。现有方法多依赖整体的视觉-语言融合,难以处理复杂、组合式描述。本文提出一种基于大语言模型(LLM)的无训练框架PARSE-VOS,实现文本与视频域间的层次化、粗到精推理。首先将自然语言查询解析为结构化语义指令;接着通过时空定位模块生成所有候选目标轨迹,由解析后的语义引导;最后通过两级推理模块识别正确目标:先由LLM进行粗粒度运动推理缩小候选范围,若仍有歧义,则触发细粒度姿态验证阶段以消除混淆。最终输出目标对象的精确分割掩码。PARSE-VOS在三个主流基准——Ref-YouTube-VOS、Ref-DAVIS17和MeViS上均取得当前最优性能。
原文摘要 · Abstract (English)
Referring Video Object Segmentation (RVOS) aims to segment an object of interest throughout a video based on a language description. The prominent challenge lies in aligning static text with dynamic visual content, particularly when objects exhibiting similar appearances with inconsistent motion and poses. However, current methods often rely on a holistic visual-language fusion that struggles with complex, compositional descriptions. In this paper, we propose \textbf{PARSE-VOS}, a novel, training-free framework powered by Large Language Models (LLMs), for a hierarchical, coarse-to-fine reasoning across text and video domains. Our approach begins by parsing the natural language query into structured semantic commands. Next, we introduce a spatio-temporal grounding module that generates all candidate trajectories for all potential target objects, guided by the parsed semantics. Finally, a hierarchical identification module select the correct target through a two-stage reasoning process: it first performs coarse-grained motion reasoning with an LLM to narrow down candidates; if ambiguity remains, a fine-grained pose verification stage is conditionally triggered to disambiguate. The final output is an accurate segmentation mask for the target object. \textbf{PARSE-VOS} achieved state-of-the-art performance on three major benchmarks: Ref-YouTube-VOS, Ref-DAVIS17, and MeViS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。