用稀疏记忆+语义提示，高效避免视频增量学习遗忘
ESSENTIAL: Episodic and Semantic Memory Integration for Video Class-Incremental Learning
- 用稀疏特征+可学习提示构建双记忆，节省存储
- 跨注意力机制从稀疏特征还原稠密时序信息
- 在多个数据集上内存大幅减少仍保持优异性能
本文针对视频类增量学习（VCIL）中的灾难性遗忘问题提出ESSENTIAL方法。现有方法多依赖存储密集时间采样片段的短期记忆，内存效率低；或仅存稀疏样本，损失关键时序信息。为此，我们设计一种结合短期记忆与语义记忆的架构：短期记忆存储稀疏特征，语义记忆通过可学习提示表示通用知识。引入交叉注意力记忆检索模块，实现从稀疏特征中恢复稠密时序表示。在TCD基准的UCF-101、HMDB51、Something-Something-V2及vCLIMB基准的UCF-101、ActivityNet、Kinetics-400上进行严格验证，结果表明，相比基线显著降低内存占用，同时取得优异性能。
原文摘要 · Abstract (English)
In this work, we tackle the problem of video classincremental learning (VCIL). Many existing VCIL methods mitigate catastrophic forgetting by rehearsal training with a few temporally dense samples stored in episodic memory, which is memory-inefficient. Alternatively, some methods store temporally sparse samples, sacrificing essential temporal information and thereby resulting in inferior performance. To address this trade-off between memory-efficiency and performance, we propose EpiSodic and SEmaNTIc memory integrAtion for video class-incremental Learning (ESSENTIAL). ESSENTIAL consists of episodic memory for storing temporally sparse features and semantic memory for storing general knowledge represented by learnable prompts. We introduce a novel memory retrieval (MR) module that integrates episodic memory and semantic prompts through cross-attention, enabling the retrieval of temporally dense features from temporally sparse features. We rigorously validate ESSENTIAL on diverse datasets: UCF-101, HMDB51, and Something-Something-V2 from the TCD benchmark and UCF-101, ActivityNet, and Kinetics-400 from the vCLIMB benchmark. Remarkably, with significantly reduced memory, ESSENTIAL achieves favorable performance on the benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。