让视觉语言模型在不访问内部结构的情况下持续学习,突破了传统方法的限制。
Black-Box Continual Learning for Vision-Language Models

- 只通过输出嵌入或逻辑值进行训练,不修改模型结构也不反向传播梯度。
- 仅用0.05百万参数就达到甚至超越白盒方法的性能,降低3000倍以上计算开销。
- 适合部署在云端、无法修改模型的现实场景,推动持续学习落地应用。
视觉语言模型(VLMs)在动态环境中快速部署,亟需持续学习能力以避免遗忘。然而,传统持续学习(CL)依赖白盒范式,难以适应云托管模型的趋势。本文提出黑盒持续学习(Black-CL)基准,模拟真实世界三大挑战:权重与架构不可见、计算受限、任务无关推理。学习者只能查询输出嵌入或逻辑值,无法进行梯度回传或结构修改。现有依赖骨干网络反向传播或参数膨胀的方法在此设置下完全失效。为此,我们提出BETA,基于仅优化文本原型的核心思路,包含三个模块:语义投影累积(SPA)实现增量知识获取,潜在分布重放(LDR)防止灾难性遗忘,测试时原型自适应(TTPA)实现动态边界调整。在十个不同数据集和多种骨干网络上的实验表明,BETA显著优于现有黑盒调优方法。令人瞩目的是,仅0.05百万可训练参数(相较竞争方法减少180–3000倍),其性能已达或超过白盒持续学习方法。我们认为Black-CL与BETA为持续学习的未来发展奠定了基础,加速了该技术从学术研究向实际系统的转化。
原文摘要 · Abstract (English)
The rapid deployment of Vision-Language Models (VLMs) in dynamic environments necessitates the ability to learn continuously without forgetting. However, traditional continual learning (CL) settings often rely on white-box paradigms, which is increasingly invalidated by the shift toward cloud-hosted models. In this paper, we introduce Black-CL, a more realistic benchmark for VLMs that enforces three primary real-world challenges: weight and architecture inaccessibility, constrained computation, and task-agnostic inference. The learner can query only output embeddings or logits, with no gradient flow through or structural modification of the backbone. Current CL methodologies, which rely on backbone backpropagation or complex parameter expansion, are fundamentally incompatible with these constraints. Under this setting, we propose BETA, a simple yet effective baseline built on the key insight that solely optimizing textual prototypes can navigate the complexities of CL. BETA integrates three core components: Semantic Projection Accumulation (SPA) for incremental knowledge acquisition, Latent Distribution Replay (LDR) for anchoring the embedding space against catastrophic forgetting, and Test-Time Prototype Adaptation (TTPA) for dynamic, instance-aware boundary refinement. Extensive experiments across ten diverse datasets and various backbones demonstrate that BETA significantly outperforms existing black-box tuners. Remarkably, with only 0.05 M trainable parameters, a 180--3000$\times$ reduction compared to competitive methods, BETA achieves performance on par with or even exceeding white-box CL methods. We believe Black-CL and BETA provide a foundational framework for future advancements in continual learning and accelerates the transition of continual learning from academia to real-world systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。