用跨模态自表达模型提升图文图像聚类,保留各自语义结构。
Learning Deep Modality-Shared Self-Expressiveness for Image Clustering with Textual Information

- 构建共享自表达模型,联合学习图像与文本的内在结构。
- 在UCF-101等3个数据集上提升超3%,聚类性能更优。
- 无需微调即可迁移至检索、零样本分类任务,适合多模态应用。
利用视觉语言模型(VLM)提取的文本信息进行图像聚类已成为重要方向。现有方法通常为每张图像匹配对应文本,通过直接强化跨模态一致性来优化多模态表示,如最大化预训练模型中继承的图像-文本相似度。然而,这种策略在未显式建模各模态内部结构的情况下对异构表示进行对齐,可能导致不可靠的对齐或破坏对聚类至关重要的模态特异性结构。本文提出一种简单而严谨的方法——深度模态共享自表达模型(DeepMORSE),通过模态共享的自表达模型发现跨模态结构,同时学习符合多个模态特定子空间并集的结构化表示。我们从理论上证明,模态共享的自表达系数能抑制类别间噪声,实现子空间保持解,并指出小批量优化过程对自表达模型引入了隐式正则化。在六个常用图像聚类基准上评估表明,DeepMORSE在UCF-101、DTD-47和ImageNet-Dogs三个数据集上性能提升超过3%。此外,我们展示了所学表示的强大可迁移性,在图像检索与零样本分类等下游任务中达到当前最优表现,且无需任何任务特定损失或后处理。代码已开源:https://github.com/mengxianghan123/DeepMORSE。
原文摘要 · Abstract (English)
Leveraging textual information for image clustering has emerged as a promising direction, largely owing to the powerful representations learned by Vision-Language Models (VLMs). Existing approaches typically retrieve a textual counterpart for each image and then refine multimodal representations by directly enforcing cross-modal agreement, e.g., maximizing image-text similarity inherited from pretrained VLMs. However, such a strategy aligns heterogeneous representations across modalities without explicitly modeling the intrinsic structure within each modality and thus might yield unreliable alignment or distort modality-specific structures that are crucial for clustering. In this paper, we propose a simple but principled approach, termed deep modality-shared self-expressive model (DeepMORSE), which discovers cross-modal structures via a modality-shared self-expressive model and simultaneously learns structured representations that conform to a union of modality-specific subspaces. Moreover, we theoretically justify that the modality-shared self-expressive coefficients suppress inter-class noise towards a subspace-preserving solution, and show that mini-batch optimization procedure introduces an implicit regularization onto the self-expressive model. We evaluate our DeepMORSE on six widely used image clustering benchmarks and observe performance improvements exceeding 3% on the UCF-101, DTD-47, and ImageNet-Dogs datasets. In addition, we demonstrate the strong transferability of the learned representations by achieving state-of-the-art performance on downstream tasks such as image retrieval and zero-shot classification---without requiring any task-specific losses or post-processing. The code is available at: https://github.com/mengxianghan123/DeepMORSE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。