用少量数据让大模型读懂任意文字标签的图像分割
Beyond-Labels: Advancing Open-Vocabulary Segmentation With Vision-Language Models
- 轻量级模块融合冻结视觉与语言特征
- 在PASCAL-5i上实现超越现有方法的精度
- 适合需要快速适配新类别标签的场景
开放词汇语义分割旨在使用训练中未见过的任意文本标签对图像中的物体进行分类和轮廓描绘。自监督学习在有效训练下可解决大量视觉与语言处理问题。本文研究了将预训练基础模型适配至开放词汇分割任务的简单高效方法。提出「Beyond-Labels」，一种基于轻量级Transformer的融合模块，仅需少量图像分割数据，即可融合冻结的视觉表征与语言概念。该策略使模型在无需大规模重训的前提下，充分利用预训练模型的广泛知识，具备数据高效性与可扩展性。此外，通过傅里叶嵌入捕捉图像位置信息，提升泛化能力，实现平滑一致的空间编码。我们在标准基准PASCAL-5i上进行了详尽的消融实验，结果表明：即使在冻结视觉与语言表示的情况下，该方法仍表现更优。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation attempts to classify and outline objects in an image using arbitrary text labels, including those unseen during training. Self-supervised learning resolves numerous visual and linguistic processing problems when effectively trained. This study investigates simple yet efficient methods for adapting previously learned foundation models for open-vocabulary semantic segmentation tasks. Our research proposes "Beyond-Labels", a lightweight transformer-based fusion module that uses a small amount of image segmentation data to fuse frozen visual representations with language concepts. This strategy allows the model to leverage the extensive knowledge of pre-trained models without requiring significant retraining, making the approach data-efficient and scalable. Furthermore, we capture positional information in images using Fourier embeddings, improving generalization and enabling smooth and consistent spatial encoding. We perform thorough ablation studies to examine the main components of our proposed method. On the standard benchmark PASCAL-5i, the method performs better despite being trained on frozen vision and language representations. Index Terms: Beyond-Labels, open-vocabulary semantic segmentation, Fourier embeddings, PASCAL-5i
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。