通过跨模态一致性自增强,让图像聚类突破预训练模型性能天花板
Self-Enhanced Image Clustering with Cross-Modal Semantic Consistency
- 先对齐图文语义,再用伪标签自优化编码器
- 在6个数据集上超越现有方法,小模型媲美大模型
- 适合追求高效高精度聚类的算法研究者
尽管CLIP等大型图文预训练模型提供了强大的通用特征,但现有方法通常冻结编码器,导致模型的通用表征与特定聚类任务需求不匹配,限制了性能上限。为突破这一瓶颈,我们提出基于跨模态语义一致性的自增强框架,实现高效图像聚类。第一阶段聚焦跨模态语义一致性,通过在实例、聚类分配和聚类中心三个层面挖掘生成的图文对一致性,训练轻量级聚类头以对齐预训练模型的丰富语义。该过程借助一种生成更高质量聚类中心的新方法及动态平衡正则化,确保聚类分配分布均匀。第二阶段引入自增强微调策略:第一阶段对齐后的模型作为可靠的伪标签生成器,其自生成的监督信号用于联合优化视觉编码器与聚类头,释放其全部潜力。在六个主流数据集上的大量实验表明,本方法显著优于现有深度聚类方法。值得注意的是,我们的ViT-B/32模型已达到甚至超过基于更大规模ViT-L/14的最先进方法的准确率。
原文摘要 · Abstract (English)
While large language-image pre-trained models like CLIP offer powerful generic features for image clustering, existing methods typically freeze the encoder. This creates a fundamental mismatch between the model's task-agnostic representations and the demands of a specific clustering task, imposing a ceiling on performance. To break this ceiling, we propose a self-enhanced framework based on cross-modal semantic consistency for efficient image clustering. Our framework first builds a strong foundation via Cross-Modal Semantic Consistency and then specializes the encoder through Self-Enhancement. In the first stage, we focus on Cross-Modal Semantic Consistency. By mining consistency between generated image-text pairs at the instance, cluster assignment, and cluster center levels, we train lightweight clustering heads to align with the rich semantics of the pre-trained model. This alignment process is bolstered by a novel method for generating higher-quality cluster centers and a dynamic balancing regularizer to ensure well-distributed assignments. In the second stage, we introduce a Self-Enhanced fine-tuning strategy. The well-aligned model from the first stage acts as a reliable pseudo-label generator. These self-generated supervisory signals are then used to feed back the efficient, joint optimization of the vision encoder and clustering heads, unlocking their full potential. Extensive experiments on six mainstream datasets show that our method outperforms existing deep clustering methods by significant margins. Notably, our ViT-B/32 model already matches or even surpasses the accuracy of state-of-the-art methods built upon the far larger ViT-L/14.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。