解决多模态模型学新数据时遗忘旧知识的问题，仅用2.4%参数达到更好效果。
TAIL: Text-Audio Incremental Learning
- 用提示调优结合相似性与特征蒸馏，减少遗忘。
- 在四个数据集上显著优于旧方法，尤其抗遗忘能力更强。
- 适合资源有限且需持续学习的多模态应用。
许多研究结合文本与音频以捕捉多模态信息，但忽视了模型在新数据集上的泛化能力。引入新数据可能影响原数据特征空间，导致灾难性遗忘。同时，大模型参数会显著影响训练性能。为此，我们提出一种新的任务——文本-音频增量学习（TAIL），并提出新方法PTAT（Prompt Tuning for Audio-Text incremental learning）。该方法利用提示调优优化模型参数，并引入音频-文本相似性与特征蒸馏模块，有效缓解灾难性遗忘。我们在AudioCaps、Clotho、BBC Sound Effects和Audioset数据集上对方法进行基准测试，结果表明，我们的方法显著优于现有增量学习方法，尤其在旧数据集上表现出更强的抗遗忘能力。相比全参数微调（Sequential）方法，本模型仅需2.42%的参数，性能却高出4.46%。
原文摘要 · Abstract (English)
Many studies combine text and audio to capture multi-modal information but they overlook the model's generalization ability on new datasets. Introducing new datasets may affect the feature space of the original dataset, leading to catastrophic forgetting. Meanwhile, large model parameters can significantly impact training performance. To address these limitations, we introduce a novel task called Text-Audio Incremental Learning (TAIL) task for text-audio retrieval, and propose a new method, PTAT, Prompt Tuning for Audio-Text incremental learning. This method utilizes prompt tuning to optimize the model parameters while incorporating an audio-text similarity and feature distillation module to effectively mitigate catastrophic forgetting. We benchmark our method and previous incremental learning methods on AudioCaps, Clotho, BBC Sound Effects and Audioset datasets, and our method outperforms previous methods significantly, particularly demonstrating stronger resistance to forgetting on older datasets. Compared to the full-parameters Finetune (Sequential) method, our model only requires 2.42\% of its parameters, achieving 4.46\% higher performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。