解决开放词汇全景分割中类别偏见与区域对齐问题，提升新类别识别能力。
Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentation
- 用CLIP条件对象性调整，缓解训练类别外物体的掩码抑制
- 通过轻量级掩码-文本优化，显著提升未见类别的定位与分类性能
- 模块化设计，内存开销小，适用于实际部署场景
开放词汇全景分割仍受两大耦合问题制约：(i) 掩码选择偏差，即在封闭词汇上训练的对象性头会抑制训练中未见类别的掩码；(ii) 视觉语言模型（如CLIP）的局部理解能力有限，其优化目标为全局图像分类而非定位分割。本文提出OVRCOAT，一种简单、模块化的框架，同时应对上述问题。首先，引入基于CLIP的物体性调整（COAT），动态更新前景/背景概率，保留未见类别的高质量掩码；其次，设计开放词汇掩码-文本精炼（OVR），强化CLIP在区域层面的对齐能力，显著提升已见与未见类别的分类精度，且内存消耗远低于以往微调方案。两者协同改进对象性估计与掩码识别，带来一致的全景分割性能提升。尽管结构简洁，OVRCOAT在ADE20K上达到5.5%的PQ提升，于Mapillary Vistas和Cityscapes分别实现7.1%和3%的提升，刷新当前最佳表现。代码已开源。
原文摘要 · Abstract (English)
Open-vocabulary panoptic segmentation remains hindered by two coupled issues: (i) mask selection bias, where objectness heads trained on closed vocabularies suppress masks of categories not observed in training, and (ii) limited regional understanding in vision-language models such as CLIP, which were optimized for global image classification rather than localized segmentation. We introduce OVRCOAT, a simple, modular framework that tackles both. First, a CLIP-conditioned objectness adjustment (COAT) updates background/foreground probabilities, preserving high-quality masks for out-of-vocabulary objects. Second, an open-vocabulary mask-to-text refinement (OVR) strengthens CLIP's region-level alignment to improve classification of both seen and unseen classes with markedly lower memory cost than prior fine-tuning schemes. The two components combine to jointly improve objectness estimation and mask recognition, yielding consistent panoptic gains. Despite its simplicity, OVRCOAT sets a new state of the art on ADE20K (+5.5% PQ) and delivers clear gains on Mapillary Vistas and Cityscapes (+7.1% and +3% PQ, respectively). The code is available at: https://github.com/nickormushev/OVRCOAT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。