用智能体动态调度工具,实现多模态视频目标分割的灵活自适应。
Towards Agentic AI for Multimodal-Guided Video Object Segmentation
- 基于大语言模型生成动态工作流,按需调用专用工具处理多模态输入。
- 在RVOS和Ref-AVS任务上优于现有方法,无需额外训练。
- 适合需要快速适配新场景的视频理解应用开发者。
基于指代的视频对象分割是一项多模态任务,需要根据外部提示生成精细的分割结果。传统方法通常依赖专门训练的模型,存在计算复杂度高和人工标注成本大的问题。视觉-语言基础模型的进展为无需训练的方法提供了新方向。已有研究利用这些通用模型实现细粒度分割,性能可媲美全监督专用模型。然而,现有方法依赖固定流程,难以适应任务的动态特性。为此,我们提出多模态智能体(Multi-Modal Agent),一种新型代理系统,以更灵活、自适应的方式解决该任务。具体而言,该方法利用大语言模型(LLM)的推理能力,为每个输入生成动态工作流,迭代调用针对不同模态设计的专用工具,识别由多模态线索描述的目标对象。该代理方法在两个多模态条件下的视频对象分割任务(RVOS和Ref-AVS)上显著优于先前方法。
原文摘要 · Abstract (English)
Referring-based Video Object Segmentation is a multimodal problem that requires producing fine-grained segmentation results guided by external cues. Traditional approaches to this task typically involve training specialized models, which come with high computational complexity and manual annotation effort. Recent advances in vision-language foundation models open a promising direction toward training-free approaches. Several studies have explored leveraging these general-purpose models for fine-grained segmentation, achieving performance comparable to that of fully supervised, task-specific models. However, existing methods rely on fixed pipelines that lack the flexibility needed to adapt to the dynamic nature of the task. To address this limitation, we propose Multi-Modal Agent, a novel agentic system designed to solve this task in a more flexible and adaptive manner. Specifically, our method leverages the reasoning capabilities of large language models (LLMs) to generate dynamic workflows tailored to each input. This adaptive procedure iteratively interacts with a set of specialized tools designed for low-level tasks across different modalities to identify the target object described by the multimodal cues. Our agentic approach demonstrates clear improvements over prior methods on two multimodal-conditioned VOS tasks: RVOS and Ref-AVS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。