用线性适配器提升图像分类中新类发现效果
Sparsity Hurts: Simple Linear Adapter Can Boost Generalized Category Discovery
- 在ViT每层插入线性适配器，替代非线性结构
- 在多个数据集上显著优于复杂基线模型
- 适合需要高效新类别识别的研究者
通用类别发现（GCD）旨在从无标签数据中识别新类别，同时保持对已见类别的分类能力。以往方法通常依赖预训练模型的可迁移表征，通过部分微调（仅更新ViT最后一层）或视觉提示调优（在输入中添加可学习向量）进行适配。但传统部分微调灵活性不足，无法调整整个模型；而视觉提示调优易过拟合，受初始化敏感且容量受限。为此，我们提出LAGCD，一种简单有效的GCD方法：在每个ViT块中嵌入残差线性适配器。从特征稀疏性角度出发，系统表明传统适配器中的非线性会损害性能，而我们的线性适配器通过增强模型灵活性提升表现。此外，引入辅助分布对齐损失以缓解已见与新类别间预测偏差问题。在通用和细粒度数据集上的大量实验表明，LAGCD持续优于多种复杂基线模型。源代码已开源。
原文摘要 · Abstract (English)
Generalized Category Discovery (GCD) seeks to identify novel categories from unlabeled data while retaining the classification ability of seen categories. Prior GCD methods commonly leverage transferable representations from pre-trained models, adapting to downstream datasets via partial fine-tuning (updating only the final ViT block) and visual prompt tuning (appending learnable vectors to inputs). However, conventional partial fine-tuning offers limited flexibility, as it fails to adapt the entire model; meanwhile, visual prompt tuning is prone to overfitting, due to its sensitivity to initialization and inherently constrained capacity. To address these limitations, we propose LAGCD, a simple yet effective GCD approach that embeds a residual linear adapter into each ViT block. From the perspective of feature sparsity, we systematically show that non-linearity in conventional adapters impairs performance, whereas our linear adapter enhances it by enabling more flexible model capacity. We further introduce an auxiliary distribution alignment loss to mitigate the negative impact of biased predictions between seen and novel categories. Extensive experiments on both generic and fine-grained datasets confirm that LAGCD consistently improves performance over many sophisticated baselines. The source code is available at https://github.com/yebo0216best/LAGCD
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。