提出新框架让模型持续学习并跨域通用,解决真实环境适应难题。
Domain Generalizable Continual Learning
- 分离语义与领域信息,动态转换表示以对齐输出
- 在多个任务和未见领域上表现更稳定,提升泛化能力
- 轻量级插件设计,兼容主流持续学习方法
为适应动态现实环境,智能系统需持续学习新技能并泛化至多样未见场景。本文提出一种新颖且贴近现实的设定——领域可泛化的持续学习(DGCL):模型按序学习各含单一领域的任务,目标是在所有已遇任务与领域中均表现良好。该设定在获取、保留和利用语义与领域相关知识方面带来独特挑战。尽管现有持续学习(CL)方法使用预训练模型(PTMs)提升任务泛化性,但通常假设每项任务训练与测试领域相同,因此在DGCL中表现不佳。为此,本文提出基于PTMs的自适应领域变换(DoT)方法。受人脑分布式加中心结构启发,DoT在表征学习中解耦语义与领域相关信息,并自适应地跨领域转换任务表示以实现输出对齐,确保预测平衡且泛化。DoT作为即插即用策略,在全参数微调与参数高效微调范式下显著提升主流CL基线在DGCL中的表现,经大量实验验证。同时,DoT能积累领域通用知识,且实现轻量化部署。
原文摘要 · Abstract (English)
To adapt effectively to dynamic real-world environments, intelligent systems must continually acquire new skills while generalizing them to diverse, unseen scenarios. Here, we introduce a novel and realistic setting named domain generalizable continual learning (DGCL): a model learns sequential tasks with each involving a single domain, aiming to perform well across all encountered tasks and domains. This setting poses unique challenges in acquiring, retaining, and leveraging both semantic- and domain-relevant information for robust generalization. Although state-of-the-art continual learning (CL) methods have employed pre-trained models (PTMs) to enhance task-specific generalization, they typically assume identical training and testing domains for each task and therefore perform poorly in DGCL. To this end, we propose adaptive Domain Transformation (DoT), an innovative PTMs-based approach tailored to DGCL. Inspired by the distributed-plus-hub theory of the human brain, DoT disentangles semantic- and domain-relevant information in representation learning, and adaptively transforms task representations across various domains for output alignment, ensuring balanced and generalized predictions. DoT serves as a plug-in strategy that greatly facilitates state-of-the-art CL baselines under both full parameter tuning and parameter-efficient tuning paradigms in DGCL, validated by extensive experiments. Also, DoT is shown to accumulate domain-generalizable knowledge from DGCL, and ensure resource efficiency with a lightweight implementation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。