无需预设类别名,自动识别并命名新物体,实现真正无词汇表语义分割。
From Open-Vocabulary to Vocabulary-Free Semantic Segmentation
- 用视觉语言模型自动识物并生成类别名,摆脱人工定义词汇限制。
- 在多个公开数据集上验证,自动化流程显著提升无词汇表分割准确率。
- 适合需要动态识别未知物体的现实场景,如自动驾驶、机器人感知。
开放词汇语义分割使模型能够识别训练数据之外的新物体类别。尽管这种灵活性是重要进展,但现有方法仍依赖手动指定的类别名称作为输入,导致实际应用中存在固有瓶颈。本文提出一种无词汇表语义分割流程,彻底消除预定义类别词汇的需求。具体而言,我们解决了一个鸡与蛋难题:用户需知晓场景中所有潜在物体才能识别它们,而分割的目的恰恰是发现这些物体。所提方法利用视觉语言模型自动识别物体并生成合适的类别名称,旨在克服类别指定与命名质量的挑战。在多个公开数据集上的大量实验表明,文本编码器在模型性能中起关键作用,尤其当图像与生成描述配对时。尽管分类文本编码器对类别标记过程中的误检敏感,增加了任务复杂性,但我们证明,完全自动化的流程在多种真实场景下显著提升了无词汇表分割的准确性。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation enables models to identify novel object categories beyond their training data. While this flexibility represents a significant advancement, current approaches still rely on manually specified class names as input, creating an inherent bottleneck in real-world applications. This work proposes a Vocabulary-Free Semantic Segmentation pipeline, eliminating the need for predefined class vocabularies. Specifically, we address the chicken-and-egg problem where users need knowledge of all potential objects within a scene to identify them, yet the purpose of segmentation is often to discover these objects. The proposed approach leverages Vision-Language Models to automatically recognize objects and generate appropriate class names, aiming to solve the challenge of class specification and naming quality. Through extensive experiments on several public datasets, we highlight the crucial role of the text encoder in model performance, particularly when the image text classes are paired with generated descriptions. Despite the challenges introduced by the sensitivity of the segmentation text encoder to false negatives within the class tagging process, which adds complexity to the task, we demonstrate that our fully automated pipeline significantly enhances vocabulary-free segmentation accuracy across diverse real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。