让分割模型边推理边查外部知识,应对实时新信息。
Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search
- 推理与外部搜索交替进行,突破大模型知识冻结限制。
- 在OK-VOS上超越现有方法,显著提升动态场景分割效果。
- 适合需要实时更新或领域专有知识的视觉任务场景。
基于语言的图像分割已成为计算机视觉中的热门课题。尽管多模态大语言模型(MLLMs)为分割系统赋予了推理能力,但其内部知识固化,难以应对需实时信息或特定领域概念的现实场景。本文提出新型分割范式Seg-ReSearch,通过实现推理与外部搜索的交替机制,使分割系统能处理超出MLLM静态知识的开放世界查询。为有效训练该能力,我们设计分层奖励机制,平衡初始引导与渐进激励,缓解稀疏结果信号与严格步骤监督之间的矛盾。为评估,我们构建了需外部知识的挑战性基准OK-VOS,用于视频对象分割。在OK-VOS及两个现有推理分割基准上的实验表明,Seg-ReSearch显著优于当前最优方法。代码与数据将公开于https://github.com/iSEE-Laboratory/Seg-ReSearch。
原文摘要 · Abstract (English)
Segmentation based on language has been a popular topic in computer vision. While recent advances in multimodal large language models (MLLMs) have endowed segmentation systems with reasoning capabilities, these efforts remain confined by the frozen internal knowledge of MLLMs, which limits their potential for real-world scenarios that involve up-to-date information or domain-specific concepts. In this work, we propose \textbf{Seg-ReSearch}, a novel segmentation paradigm that overcomes the knowledge bottleneck of existing approaches. By enabling interleaved reasoning and external search, Seg-ReSearch empowers segmentation systems to handle dynamic, open-world queries that extend beyond the frozen knowledge of MLLMs. To effectively train this capability, we introduce a hierarchical reward design that harmonizes initial guidance with progressive incentives, mitigating the dilemma between sparse outcome signals and rigid step-wise supervision. For evaluation, we construct OK-VOS, a challenging benchmark that explicitly requires outside knowledge for video object segmentation. Experiments on OK-VOS and two existing reasoning segmentation benchmarks demonstrate that our Seg-ReSearch improves state-of-the-art approaches by a substantial margin. Code and data will be released at https://github.com/iSEE-Laboratory/Seg-ReSearch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。