用语义概念融合图像,高效发现未知类别
SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery
- 以文本概念为基底,构建跨模态统一表示
- 跨模态协方差筛选关键概念,提升表征质量
- 计算量小,适合资源受限场景下的新类别发现
广义类别发现(GCD)旨在从无标签数据中识别新类别,同时利用少量已知类别的有标签样本。仅基于图像特征训练参数分类器易对旧类别过拟合,近期多模态方法通过引入文本信息提升性能,但常独立处理模态且计算开销大。本文提出SpectralGCD,一种高效多模态GCD方法,利用CLIP模型的图像-概念交叉相似性构建统一跨模态表示。每张图像被表达为大规模任务无关概念词典中语义概念的混合,使学习锚定于明确语义,减少对虚假视觉线索的依赖。为保持轻量学生模型所学表示的语义质量,提出谱过滤机制,通过强教师模型测量的软似然交叉模态协方差矩阵,自动保留词典中相关概念。正向与反向知识蒸馏确保学生模型的跨模态表示兼具语义充分性与对齐性。在六个基准上,SpectralGCD性能达到或显著优于当前最优方法,且计算成本仅为后者的几分之一。代码已公开:https://github.com/miccunifi/SpectralGCD。
原文摘要 · Abstract (English)
Generalized Category Discovery (GCD) aims to identify novel categories in unlabeled data while leveraging a small labeled subset of known classes. Training a parametric classifier solely on image features often leads to overfitting to old classes, and recent multimodal approaches improve performance by incorporating textual information. However, they treat modalities independently and incur high computational cost. We propose SpectralGCD, an efficient and effective multimodal approach to GCD that uses CLIP cross-modal image-concept similarities as a unified cross-modal representation. Each image is expressed as a mixture over semantic concepts from a large task-agnostic dictionary, which anchors learning to explicit semantics and reduces reliance on spurious visual cues. To maintain the semantic quality of representations learned by an efficient student, we introduce Spectral Filtering which exploits a cross-modal covariance matrix over the softmaxed similarities measured by a strong teacher model to automatically retain only relevant concepts from the dictionary. Forward and reverse knowledge distillation from the same teacher ensures that the cross-modal representations of the student remain both semantically sufficient and well-aligned. Across six benchmarks, SpectralGCD delivers accuracy comparable to or significantly superior to state-of-the-art methods at a fraction of the computational cost. The code is publicly available at: https://github.com/miccunifi/SpectralGCD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。