用语言描述辅助少样本增量学习，让模型像人一样理解新知识。
Language-Inspired Relation Transfer for Few-shot Class-Incremental Learning
- 结合视觉与文本信息，通过图关系迁移实现跨模态知识传递。
- 在mini-ImageNet和CIFAR-100上分别提升13%和7%的最终性能。
- 适合需要持续学习新类别的场景，如开放世界图像识别。
通过少量样本和语言描述来识别新类别是人类学习系统的本质能力。这种终身学习能力有助于在开放世界中区分新旧知识，即少样本增量学习（FSCIL）。现有方法主要依赖对视觉编码器的精细调优，导致基础知识与增量知识之间存在明显权衡。受人类学习启发，我们提出一种语言引导的关系迁移（LRT）范式，通过联合视觉线索与文本描述来理解物体，包含两个核心步骤：首先，通过图关系转换模块将预训练文本知识迁移到视觉领域；其次，利用文本-视觉原型融合模块融合视觉与语言嵌入。为缓解视觉微调带来的域偏移问题，提出上下文提示学习以快速对齐域特征，并采用想象对比学习缓解对齐过程中文本数据不足的问题。通过域对齐与图文迁移的协同学习，LRT在mini-ImageNet和CIFAR-100 FSCIL基准上的最终会话性能分别超越当前最优模型13%和7%。
原文摘要 · Abstract (English)
Depicting novel classes with language descriptions by observing few-shot samples is inherent in human-learning systems. This lifelong learning capability helps to distinguish new knowledge from old ones through the increase of open-world learning, namely Few-Shot Class-Incremental Learning (FSCIL). Existing works to solve this problem mainly rely on the careful tuning of visual encoders, which shows an evident trade-off between the base knowledge and incremental ones. Motivated by human learning systems, we propose a new Language-inspired Relation Transfer (LRT) paradigm to understand objects by joint visual clues and text depictions, composed of two major steps. We first transfer the pretrained text knowledge to the visual domains by proposing a graph relation transformation module and then fuse the visual and language embedding by a text-vision prototypical fusion module. Second, to mitigate the domain gap caused by visual finetuning, we propose context prompt learning for fast domain alignment and imagined contrastive learning to alleviate the insufficient text data during alignment. With collaborative learning of domain alignments and text-image transfer, our proposed LRT outperforms the state-of-the-art models by over $13\%$ and $7\%$ on the final session of mini-ImageNet and CIFAR-100 FSCIL benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。