通过分离特定与通用知识,提升持续文本分类的遗忘抑制能力
Information-Theoretic Complementary Prompts for Improved Continual Text Classification
- 设计双提示空间:私有提示存任务特异性知识,共享提示存通用知识
- 在多个基准上超越现有方法,显著减少灾难性遗忘并提升知识迁移
- 适合需要长期学习新任务且无数据重放条件的研究与应用
持续文本分类(CTC)旨在随时间不断对新文本进行分类,同时最小化对已有知识的灾难性遗忘。现有方法多关注任务特定知识,忽视了共享的、任务无关知识的重要性。受互补学习系统理论启发,我们提出信息论互补提示(InfoComp),显式学习两个独立的提示空间:私有提示(P-Prompt)和共享提示(S-Prompt),分别编码任务特定与任务不变知识,使模型无需依赖数据回放即可顺序学习分类任务。为促进更有效的提示学习,InfoComp采用信息论框架,最大化不同参数(或编码表示)间的互信息。在此框架下,设计两种新损失函数:(1) 强化 P-Prompt 中任务特定知识的积累,有效缓解遗忘;(2) 提升 S-Prompt 中任务不变知识的保留,增强前向知识迁移。在多个 CTC 基准上的实验表明,该方法优于此前最先进方法。
原文摘要 · Abstract (English)
Continual Text Classification (CTC) aims to continuously classify new text data over time while minimizing catastrophic forgetting of previously acquired knowledge. However, existing methods often focus on task-specific knowledge, overlooking the importance of shared, task-agnostic knowledge. Inspired by the complementary learning systems theory, which posits that humans learn continually through the interaction of two systems -- the hippocampus, responsible for forming distinct representations of specific experiences, and the neocortex, which extracts more general and transferable representations from past experiences -- we introduce Information-Theoretic Complementary Prompts (InfoComp), a novel approach for CTC. InfoComp explicitly learns two distinct prompt spaces: P(rivate)-Prompt and S(hared)-Prompt. These respectively encode task-specific and task-invariant knowledge, enabling models to sequentially learn classification tasks without relying on data replay. To promote more informative prompt learning, InfoComp uses an information-theoretic framework that maximizes mutual information between different parameters (or encoded representations). Within this framework, we design two novel loss functions: (1) to strengthen the accumulation of task-specific knowledge in P-Prompt, effectively mitigating catastrophic forgetting, and (2) to enhance the retention of task-invariant knowledge in S-Prompt, improving forward knowledge transfer. Extensive experiments on diverse CTC benchmarks show that our approach outperforms previous state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。