arXiv:2512.09172cs.CVcs.AI2025-12被引 1

让视觉语言模型持续学习新组合,不遗忘旧知识。

Prompt-Based Continual Compositional Zero-Shot Learning

  • 用提示词+多教师蒸馏,防止知识遗忘。
  • 新属性与对象嵌入互不重叠,提升区分度。
  • 适合需要持续学习新组合的AI系统使用。

我们解决视觉语言模型在组合零样本学习(CZSL)中对新属性、新物体及其组合的持续适应问题,同时避免对已有知识的遗忘。与类别互不重叠的经典持续学习不同,连续组合零样本学习(CCZSL)更复杂:属性和物体可能跨会话重复出现,而组合唯一。基于冻结的VLM主干网络,我们提出首个提示词驱动的连续组合零样本学习(PromptCCZSL)框架,通过近期加权多教师蒸馏保留先验知识。该框架采用会话感知的组合提示词融合多模态特征以生成新组合表示,而属性与物体提示词则通过会话无关融合保持全局语义一致性,并由余弦锚定损失(CAL)进一步稳定先验知识。为增强当前会话的适应能力,正交投影损失(OPL)确保新嵌入与旧嵌入不重叠,内会话多样性损失(IDL)促进当前嵌入间的差异性,生成更丰富且判别性强的表征。我们还引入综合评估协议,联合衡量灾难性遗忘与组合泛化能力。在UT-Zappos与C-GQA基准上的大量实验表明,PromptCCZSL显著优于现有VLM与非VLM基线,在封闭世界设置下树立了新的性能基准。

原文摘要 · Abstract (English)

We tackle continual adaptation of vision-language models to new attributes, objects, and their compositions in Compositional Zero-Shot Learning (CZSL), while preventing forgetting of prior knowledge. Unlike classical continual learning where classes are disjoint, CCZSL is more complex as attributes and objects may reoccur across sessions while compositions remain unique. Built on a frozen VLM backbone, we propose the first Prompt-based Continual Compositional Zero-Shot Learning (PromptCCZSL) framework that retains prior knowledge through recency-weighted multi-teacher distillation. It employs session-aware compositional prompts to fuse multimodal features for new compositions, while attribute and object prompts are learned through session-agnostic fusion to maintain global semantic consistency, which is further stabilized by a Cosine Anchor Loss (CAL) to preserve prior knowledge. To enhance adaptation in the current session, an Orthogonal Projection Loss (OPL) ensures that new attribute and object embeddings remain distinct from previous ones, preventing overlap, while an Intra-Session Diversity Loss (IDL) promotes variation among current-session embeddings for richer, more discriminative representations. We also introduce a comprehensive protocol that jointly measures catastrophic forgetting and compositional generalization. Extensive experiments on UT-Zappos and C-GQA benchmarks demonstrate that PromptCCZSL achieves substantial improvements over prior VLM-based and non-VLM baselines, setting a new benchmark for CCZSL in closed-world settings.

零样本学习持续学习视觉语言模型提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。