arXiv:2604.14147cs.CV2026-04被引 2

让分割模型学会识别新出现的物体,靠实时网络信息增强理解。

ROSE: Retrieval-Oriented Segmentation Enhancement

论文配图:ROSE: Retrieval-Oriented Segmentation Enhancement
图 1 · 摘自论文原文
  • 用网络检索获取实时信息,补足模型对新事物的知识空白。
  • 在新闻类数据集上,分割精度比基线高19.2个gIoU点。
  • 适配任意多模态大模型,无需重新训练,适合快速部署。

基于多模态大语言模型(MLLM)的分割模型(如LISA)常因无法获取最新知识,难以识别新出现或未见实体。为此,我们提出新型分割任务——新颖与新兴实体分割任务(NEST),聚焦于两类问题:(i) 模型训练中未出现的新实体;(ii) 虽在模型知识范围内但需实时外部信息才能准确识别的新兴实体。为支持该任务研究,我们构建了基于自动化流水线的NEST基准数据集,生成大量新闻相关样本以实现全面评估。同时,提出一种即插即用框架ROSE:检索导向分割增强。其包含四个核心组件:首先,互联网检索增强生成模块,利用用户提供的多模态输入检索实时网络信息;其次,文本提示增强器,注入最新信息与背景知识,提升模型对新兴实体的感知能力;再次,视觉提示增强器,通过网络图像弥补模型对新实体的视觉缺失;最后,WebSense模块智能判断是否触发检索,保障效率。实验表明,ROSE在NEST基准上显著提升性能,相较强基线Gemini-2.0 Flash的检索方法,平均交并比(gIoU)提升19.2个百分点。

原文摘要 · Abstract (English)

Existing segmentation models based on multimodal large language models (MLLMs), such as LISA, often struggle with novel or emerging entities due to their inability to incorporate up-to-date knowledge. To address this challenge, we introduce the Novel Emerging Segmentation Task (NEST), which focuses on segmenting (i) novel entities that MLLMs fail to recognize due to their absence from training data, and (ii) emerging entities that exist within the model's knowledge but demand up-to-date external information for accurate recognition. To support the study of NEST, we construct a NEST benchmark using an automated pipeline that generates news-related data samples for comprehensive evaluation. Additionally, we propose ROSE: Retrieval-Oriented Segmentation Enhancement, a plug-and-play framework designed to augment any MLLM-based segmentation model. ROSE comprises four key components. First, an Internet Retrieval-Augmented Generation module is introduced to employ user-provided multimodal inputs to retrieve real-time web information. Then, a Textual Prompt Enhancer enriches the model with up-to-date information and rich background knowledge, improving the model's perception ability for emerging entities. Furthermore, a Visual Prompt Enhancer is proposed to compensate for MLLMs' lack of exposure to novel entities by leveraging internet-sourced images. To maintain efficiency, a WebSense module is introduced to intelligently decide when to invoke retrieval mechanisms based on user input. Experimental results demonstrate that ROSE significantly boosts performance on the NEST benchmark, outperforming a strong Gemini-2.0 Flash-based retrieval baseline by 19.2 in gIoU.

图像分割实时增强多模态知识更新

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。