用知识图谱增强多模态模型,缓解持续学习中的遗忘问题。
Knowledge Graph Enhanced Generative Multi-modal Models for Class-Incremental Learning
- 构建动态知识图谱,利用类别间关系增强标签表征
- 测试时通过图谱关系定位类别,减少旧类细节丢失
- 在常规和少样本持续学习中均达领先效果
计算机视觉中的持续学习面临灾难性遗忘的挑战,模型在学习新任务时难以保留旧知识。尽管现有方法尝试利用预训练模型的泛化能力减轻当前任务过拟合,模型仍会逐渐遗忘已学类别的细节,导致误分类。为此,我们提出一种知识图谱增强的生成式多模态模型(KG-GMM),在学习过程中构建动态演化的知识图谱。该方法利用图谱中的关系来扩充类别标签,并为相似类别分配不同关系以增强区分度。测试时,提出知识图谱增强推理方法,通过分析生成文本中的关系来定位具体类别,从而在学习新知识的同时减少旧类别细节的损失,缓解遗忘问题。实验表明,该方法有效利用关系信息纠正误判,在传统持续学习(CIL)和少样本持续学习(few-shot CIL)设置下均达到先进水平,验证了知识图谱在持续学习中保存知识的有效性。
原文摘要 · Abstract (English)
Continual learning in computer vision faces the critical challenge of catastrophic forgetting, where models struggle to retain prior knowledge while adapting to new tasks. Although recent studies have attempted to leverage the generalization capabilities of pre-trained models to mitigate overfitting on current tasks, models still tend to forget details of previously learned categories as tasks progress, leading to misclassification. To address these limitations, we introduce a novel Knowledge Graph Enhanced Generative Multi-modal model (KG-GMM) that builds an evolving knowledge graph throughout the learning process. Our approach utilizes relationships within the knowledge graph to augment the class labels and assigns different relations to similar categories to enhance model differentiation. During testing, we propose a Knowledge Graph Augmented Inference method that locates specific categories by analyzing relationships within the generated text, thereby reducing the loss of detailed information about old classes when learning new knowledge and alleviating forgetting. Experiments demonstrate that our method effectively leverages relational information to help the model correct mispredictions, achieving state-of-the-art results in both conventional CIL and few-shot CIL settings, confirming the efficacy of knowledge graphs at preserving knowledge in the continual learning scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。