用检索增强实现任意类别全景分割，提升未知类别的识别准确率。
Open Vocabulary Panoptic Segmentation With Retrieval Augmentation
- 通过图像-文本对构建掩码区域特征库，推理时检索匹配特征
- 在ADE20k上达到30.9 PQ、44.0 mIoU，比基线提升10.0 mIoU
- 适合需要支持开放词汇分割的视觉系统开发者使用
给定输入图像和类别名称集合，全景分割需为图像中每个像素分配类别标签和实例标签。开放词汇全景分割旨在根据用户输入对任意类别进行分割。现有方法在训练数据外的未见类别上泛化能力差。本文提出RetCLIP，一种基于检索增强的全景分割方法，通过配对图像-文本数据构建掩码区域特征数据库。推理时，以输入图像中的掩码区域特征为查询，从数据库中检索相似特征及其关联类别标签，依据相似度分配分类得分，并与CLIP得分融合生成最终结果。该方法结合此前最优方法FC-CLIP，在COCO上训练后于ADE20k测试，取得30.9 PQ、19.3 mAP、44.0 mIoU，相比基线提升4.5 PQ、2.5 mAP、10.0 mIoU。
原文摘要 · Abstract (English)
Given an input image and set of class names, panoptic segmentation aims to label each pixel in an image with class labels and instance labels. In comparison, Open Vocabulary Panoptic Segmentation aims to facilitate the segmentation of arbitrary classes according to user input. The challenge is that a panoptic segmentation system trained on a particular dataset typically does not generalize well to unseen classes beyond the training data. In this work, we propose RetCLIP, a retrieval-augmented panoptic segmentation method that improves the performance of unseen classes. In particular, we construct a masked segment feature database using paired image-text data. At inference time, we use masked segment features from the input image as query keys to retrieve similar features and associated class labels from the database. Classification scores for the masked segment are assigned based on the similarity between query features and retrieved features. The retrieval-based classification scores are combined with CLIP-based scores to produce the final output. We incorporate our solution with a previous SOTA method (FC-CLIP). When trained on COCO, the proposed method demonstrates 30.9 PQ, 19.3 mAP, 44.0 mIoU on the ADE20k dataset, achieving +4.5 PQ, +2.5 mAP, +10.0 mIoU absolute improvement over the baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。