用CLIP提取通用属性,无须记忆样本即可持续适应新领域。
Cross-Domain Attribute Alignment with CLIP: A Rehearsal-Free Approach for Class-Incremental Unsupervised Domain Adaptation
- 通过CLIP提取视觉-文本对齐的通用属性,构建跨域知识库。
- 在三个基准上超越现有方法,避免灾难性遗忘且无需存储历史数据。
- 适合需要持续学习、无记忆负担的领域自适应场景。
类增量无监督域适应(CI-UDA)旨在将模型从有标签源域迁移到无标签目标域,其中不同时间步的目标类别互不重叠且为源类子集。核心挑战在于持续缓解域偏移的同时避免对旧类别知识的灾难性遗忘。以往方法需同时维护历史目标样本用于回放,并仅对跨域共享类进行对齐,导致内存持续增长且对称对齐缺失引发遗忘。本文提出一种无回放方法:利用CLIP提取并保留与类别无关的域不变属性,构建“键值对”表示,其中键为视觉原型,值为文本提示。维护两个对应不同域的属性字典,通过促进视觉注意力一致性和预测一致性实现跨域属性对齐。该机制有效减少知识遗忘并缓解域偏移。在三个CI-UDA基准上实验表明,本方法优于当前最优模型,且无需记忆机制。代码已开源:https://github.com/RyunMi/VisTA。
原文摘要 · Abstract (English)
Class-Incremental Unsupervised Domain Adaptation (CI-UDA) aims to adapt a model from a labeled source domain to an unlabeled target domain, where the sets of potential target classes appearing at different time steps are disjoint and are subsets of the source classes. The key to solving this problem lies in avoiding catastrophic forgetting of knowledge about previous target classes during continuously mitigating the domain shift. Most previous works cumbersomely combine two technical components. On one hand, they need to store and utilize rehearsal target sample from previous time steps to avoid catastrophic forgetting; on the other hand, they perform alignment only between classes shared across domains at each time step. Consequently, the memory will continuously increase and the asymmetric alignment may inevitably result in knowledge forgetting. In this paper, we propose to mine and preserve domain-invariant and class-agnostic knowledge to facilitate the CI-UDA task. Specifically, via using CLIP, we extract the class-agnostic properties which we name as "attribute". In our framework, we learn a "key-value" pair to represent an attribute, where the key corresponds to the visual prototype and the value is the textual prompt. We maintain two attribute dictionaries, each corresponding to a different domain. Then we perform attribute alignment across domains to mitigate the domain shift, via encouraging visual attention consistency and prediction consistency. Through attribute modeling and cross-domain alignment, we effectively reduce catastrophic knowledge forgetting while mitigating the domain shift, in a rehearsal-free way. Experiments on three CI-UDA benchmarks demonstrate that our method outperforms previous state-of-the-art methods and effectively alleviates catastrophic forgetting. Code is available at https://github.com/RyunMi/VisTA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。