arXiv:2604.14762cs.CV2026-04中稿 · CVPR

提出跨模态通用类别发现方法,一次训练即可零样本跨16数据集识别新旧类别。

OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism

论文配图:OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism
图 1 · 摘自论文原文
  • 用多模态编码器提取特征,构建统一的类别发现潜在空间。
  • 零样本测试下在4类模态上平均提升准确率6.2至17.9个百分点。
  • 无需微调,适合希望解耦表示学习与类别发现的研究者。

通用类别发现(GCD)要求模型在部分标注数据下识别已知和未知类别,模拟人类分类学习。现有方法局限于单一模态且需针对数据集微调。本文提出受人脑抽象类别形成启发的模态无关GCD方法OmniGCD:使用视觉、音频、文本、遥感等模态编码器处理输入,经降维构建GCD潜在空间,并通过在合成数据上训练的Transformer模型,在测试时转换为更适于聚类的表示。为评估该方法,引入零样本GCD设置,禁止任何数据集微调。仅在合成数据上训练一次,OmniGCD即可在16个涵盖四类模态的数据集上实现零样本GCD,对已知与未知类别分类准确率显著优于基线(视觉+6.2、文本+17.9、音频+1.5、遥感+12.7百分点)。结果凸显强编码器的重要性,同时解耦了表示学习与类别发现。推动模态无关方法发展,可独立优化编码器。本工作为未来模态无关GCD研究提供基准,迈向可扩展的人类式类别发现。代码开源:https://github.com/Jordan-HS/OmniGCD

原文摘要 · Abstract (English)

Generalized Category Discovery (GCD) challenges methods to identify known and novel classes using partially labeled data, mirroring human category learning. Unlike prior GCD methods, which operate within a single modality and require dataset-specific fine-tuning, we propose a modality-agnostic GCD approach inspired by the human brain's abstract category formation. Our $\textbf{OmniGCD}$ leverages modality-specific encoders (e.g., vision, audio, text, remote sensing) to process inputs, followed by dimension reduction to construct a $\textbf{GCD latent space}$, which is transformed at test-time into a representation better suited for clustering using a novel synthetically trained Transformer-based model. To evaluate OmniGCD, we introduce a $\textbf{zero-shot GCD setting}$ where no dataset-specific fine-tuning is allowed, enabling modality-agnostic category discovery. $\textbf{Trained once on synthetic data}$, OmniGCD performs zero-shot GCD across 16 datasets spanning four modalities, improving classification accuracy for known and novel classes over baselines (average percentage point improvement of $\textbf{+6.2}$, $\textbf{+17.9}$, $\textbf{+1.5}$ and $\textbf{+12.7}$ for vision, text, audio and remote sensing). This highlights the importance of strong encoders while decoupling representation learning from category discovery. Improving modality-agnostic methods will propagate across modalities, enabling encoder development independent of GCD. Our work serves as a benchmark for future modality-agnostic GCD works, paving the way for scalable, human-inspired category discovery. All code is available $\href{https://github.com/Jordan-HS/OmniGCD}{here}$

类别发现跨模态零样本生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。