arXiv:2411.17475cs.CV2024-11IJCV被引 4

解决脑影像视觉理解中的遗忘问题,实现持续学习。

COBRA: A Continual Learning Approach to Vision-Brain Understanding

  • 设计三模块架构,分离共享与个体脑-视觉模式。
  • 在持续学习中保持旧知识,重建精度优于现有方法。
  • 适合研究脑科学与神经影像的持续学习方向者。

视觉-脑理解(VBU)旨在从功能磁共振成像(fMRI)记录的脑活动数据中提取人类感知的视觉信息。尽管近年取得显著进展,现有研究仍面临灾难性遗忘问题:模型在适应新受试者时会丢失对先前受试者的知识。本文提出一种名为COBRA的新框架,专为解决VBU中的持续学习挑战而设计。其包含三个创新模块:主题共性(SC)模块捕捉跨受试者的共享脑-视觉模式,防止遗忘;基于提示的主体特异性(PSS)模块学习每个受试者的独特脑-视觉关联;以及基于Transformer的fMRI处理模块——MRIFormer,用于融合共性和特异性模式进行重建。在持续学习设置中,仅更新新受试者的PSS和MRIFormer模块,旧模块保持不变。实验表明,COBRA有效缓解了灾难性遗忘,在持续学习与视觉-脑重建任务中均达到当前最优性能,超越已有方法。

原文摘要 · Abstract (English)

Vision-Brain Understanding (VBU) aims to extract visual information perceived by humans from brain activity recorded through functional Magnetic Resonance Imaging (fMRI). Despite notable advancements in recent years, existing studies in VBU continue to face the challenge of catastrophic forgetting, where models lose knowledge from prior subjects as they adapt to new ones. Addressing continual learning in this field is, therefore, essential. This paper introduces a novel framework called Continual Learning for Vision-Brain (COBRA) to address continual learning in VBU. Our approach includes three novel modules: a Subject Commonality (SC) module, a Prompt-based Subject Specific (PSS) module, and a transformer-based module for fMRI, denoted as MRIFormer module. The SC module captures shared vision-brain patterns across subjects, preserving this knowledge as the model encounters new subjects, thereby reducing the impact of catastrophic forgetting. On the other hand, the PSS module learns unique vision-brain patterns specific to each subject. Finally, the MRIFormer module contains a transformer encoder and decoder that learns the fMRI features for VBU from common and specific patterns. In a continual learning setup, COBRA is trained in new PSS and MRIFormer modules for new subjects, leaving the modules of previous subjects unaffected. As a result, COBRA effectively addresses catastrophic forgetting and achieves state-of-the-art performance in both continual learning and vision-brain reconstruction tasks, surpassing previous methods.

脑机接口持续学习视觉理解fMRI重建

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。