不训练模型，仅靠高质量数据实现开放词汇语义分割
ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentation
- 构建高质量图文匹配的参考数据集，提升分割精度
- 在10个基准数据集上超越所有现有无训练方法
- 适合关注数据质量、无需微调的视觉理解研究者
无训练开放词汇语义分割旨在无需模型微调的情况下，对任意文本类别进行图像分割。现有方法多依赖预训练模型（如CLIP）的注意力机制，或通过生成合成数据并设计复杂检索流程来实现。然而，其性能受限于模型能力或参考集质量不佳。本文聚焦这一任务中常被忽视的数据质量问题，发现高质量参考集能显著提升无训练开放词汇分割效果。为此提出数据驱动框架ReME，包含构建图文嵌入精准配对的参考集数据流水线，以及基于相似度的简单检索机制，充分释放数据潜力。大量实验在10个基准数据集上验证，本方法优于所有现有无训练方法，凸显数据中心设计对推进无训练开放词汇分割的关键作用。代码已开源。
原文摘要 · Abstract (English)
Training-free open-vocabulary semantic segmentation (OVS) aims to segment images given a set of arbitrary textual categories without costly model fine-tuning. Existing solutions often explore attention mechanisms of pre-trained models, such as CLIP, or generate synthetic data and design complex retrieval processes to perform OVS. However, their performance is limited by the capability of reliant models or the suboptimal quality of reference sets. In this work, we investigate the largely overlooked data quality problem for this challenging dense scene understanding task, and identify that a high-quality reference set can significantly benefit training-free OVS. With this observation, we introduce a data-quality-oriented framework, comprising a data pipeline to construct a reference set with well-paired segment-text embeddings and a simple similarity-based retrieval to unveil the essential effect of data. Remarkably, extensive evaluations on ten benchmark datasets demonstrate that our method outperforms all existing training-free OVS approaches, highlighting the importance of data-centric design for advancing OVS without training. Our code is available at https://github.com/xiweix/ReME .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。