不增加参数的CLIP增量学习方法,有效防止遗忘并提升多模态融合
BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning
- 仅在CLIP桥接层更新参数,避免新增模块和推理开销
- 通过正交低秩融合机制,在不重放数据下稳定积累知识
- 结合文本与视觉原型,适合资源受限的持续学习场景
类别增量学习(CIL)旨在持续学习新类别而不遗忘旧知识。视觉语言模型如CLIP通过多模态监督提供强大的可迁移表征,适用于CIL。然而将CLIP应用于CIL面临两大挑战:(1) 适应下游任务常需额外可学习模块,增加模型复杂度且易遗忘;(2) 多模态表征虽具互补优势,但现有方法未能充分实现视觉与文本模态的有效融合。为此,我们提出BOFA(桥接层正交低秩融合适配框架),将所有模型适配限制在CLIP现有的跨模态桥接层内,不引入额外参数或推理成本。为防止该层遗忘,采用正交低秩融合机制,将参数更新约束于数学上与历史任务特征正交的低秩“安全子空间”。这确保了无需数据回放的知识稳定累积。此外,BOFA使用跨模态混合原型,融合稳定的文本原型与由稳定适配桥接层生成的视觉原型,提升分类性能。在标准基准上的大量实验表明,相比现有方法,BOFA在准确率与效率上均表现更优。
原文摘要 · Abstract (English)
Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge. Vision-language models such as CLIP offer strong transferable representations via multi-modal supervision, making them promising for CIL. However, applying CLIP to CIL poses two major challenges: (1) adapting to downstream tasks often requires additional learnable modules, increasing model complexity and susceptibility to forgetting; and (2) while multi-modal representations offer complementary strengths, existing methods have yet to fully realize their potential in effectively integrating visual and textual modalities. To address these issues, we propose BOFA (Bridge-layer Orthogonal Fusion for Adaptation), a novel framework for CIL. BOFA confines all model adaptation exclusively to CLIP's existing cross-modal bridge-layer, thereby adding no extra parameters or inference cost. To prevent forgetting within this layer, it leverages Orthogonal Low-Rank Fusion, a mechanism that constrains parameter updates to a low-rank ``safe subspace" mathematically constructed to be orthogonal to past task features. This ensures stable knowledge accumulation without data replay. Furthermore, BOFA employs a cross-modal hybrid prototype that synergizes stable textual prototypes with visual counterparts derived from our stably adapted bridge-layer, enhancing classification performance. Extensive experiments on standard benchmarks show that BOFA achieves superior accuracy and efficiency compared to existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。