用分层文本知识提升图像聚类,让相似外观的图片也能分清语义差异。
Hierarchical Textual Knowledge for Enhanced Image Clustering

- 用大模型构建概念-属性层级知识图谱,提炼文本深层语义。
- 在20个数据集上优于现有方法,14个数据集超过零样本CLIP。
- 避免直接用文本标签导致性能下降,兼顾准确与稳定。
图像聚类旨在无监督地分组图像。传统方法仅依赖视觉信息,难以区分外观相似但语义不同的类别。近期视觉语言模型使利用文本知识增强聚类成为可能,但多数方法仅使用粗粒度类别标签或简单名词,忽略了文本空间中丰富的概念与属性级语义。本文提出知识增强聚类(KEC)方法,借助大语言模型(LLM)构建分层概念-属性结构化知识,指导聚类过程。首先将冗余文本标签压缩为抽象概念,再通过结构化提示自动提取每个概念及其相似概念对的判别性属性。该知识被实例化到每张输入图像,生成知识增强特征。结合原始视觉特征后,适配多种下游聚类算法。在20个多样化数据集上评估表明,引入额外文本知识后,KEC表现持续优于现有方法;无需训练即在14/20个数据集上超越零样本CLIP。此外,直接使用文本知识可能损害聚类效果,而KEC同时提升准确率与鲁棒性。
原文摘要 · Abstract (English)
Image clustering aims to group images in an unsupervised fashion. Traditional methods focus on knowledge from visual space, making it difficult to distinguish between visually similar but semantically different classes. Recent advances in vision-language models enable the use of textual knowledge to enhance image clustering. However, most existing methods rely on coarse class labels or simple nouns, overlooking the rich conceptual and attribute-level semantics embedded in textual space. In this paper, we propose a knowledge-enhanced clustering (KEC) method that constructs a hierarchical concept-attribute structured knowledge with the help of large language models (LLMs) to guide clustering. Specifically, we first condense redundant textual labels into abstract concepts and then automatically extract discriminative attributes for each single concept and similar concept pairs, via structured prompts to LLMs. This knowledge is instantiated for each input image to achieve the knowledge-enhanced features. The knowledge-enhanced features with original visual features are adapted to various downstream clustering algorithms. We evaluate KEC on 20 diverse datasets, showing consistent improvements across existing methods using additional textual knowledge. KEC without training outperforms zero-shot CLIP on 14 out of 20 datasets. Furthermore, the naive use of textual knowledge may harm clustering performance, while KEC provides both accuracy and robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。