不训练提升图像分割性能，融合图文与结构优化。
ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancements
- 在ViT末层改架构并融合中间层注意力图增强语义理解。
- 通过数据增强丰富图像表征，提升模型对细节的捕捉能力。
- 用大语言模型生成类别定义，释放CLIP的开放词汇潜力。
近年来，基础视觉语言模型（VLMs）重塑了计算机视觉任务的评估范式。以CLIP为代表的模型加速了开放词汇视觉任务的研究，包括开放词汇语义分割（OVSS）。尽管初期成果令人鼓舞，但现有VLM在密集预测任务上的表现仍有提升空间。本文提出一种无需训练的方法ITACLIP，通过三方面改进：1）调整ViT末层结构，并融合中间层注意力图与末层特征；2）应用数据增强策略丰富输入图像表示；3）利用大语言模型（LLMs）生成类别名称的定义与同义词，以强化CLIP的开放词汇能力。实验表明，ITACLIP在COCO-Stuff、COCO-Object、Pascal Context和Pascal VOC等分割基准上超越当前最先进方法。代码已开源。
原文摘要 · Abstract (English)
Recent advances in foundational Vision Language Models (VLMs) have reshaped the evaluation paradigm in computer vision tasks. These foundational models, especially CLIP, have accelerated research in open-vocabulary computer vision tasks, including Open-Vocabulary Semantic Segmentation (OVSS). Although the initial results are promising, the dense prediction capabilities of VLMs still require further improvement. In this study, we enhance the semantic segmentation performance of CLIP by introducing new modules and modifications: 1) architectural changes in the last layer of ViT and the incorporation of attention maps from the middle layers with the last layer, 2) Image Engineering: applying data augmentations to enrich input image representations, and 3) using Large Language Models (LLMs) to generate definitions and synonyms for each class name to leverage CLIP's open-vocabulary capabilities. Our training-free method, ITACLIP, outperforms current state-of-the-art approaches on segmentation benchmarks such as COCO-Stuff, COCO-Object, Pascal Context, and Pascal VOC. Our code is available at https://github.com/m-arda-aydn/ITACLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。