提出AVReCAP方法,在不遗忘源模型的前提下,持续适应音频视觉新场景。
Audio-Visual Continual Test-Time Adaptation without Forgetting
- 只更新模态融合层参数,实现跨任务知识迁移。
- 在多类噪声数据上性能超越现有方法,遗忘率显著降低。
- 适合需要长期在线学习的多模态系统部署。
音视频持续测试时自适应需在无标签非平稳域中持续调整源音视频模型,其中任一或双模态分布可能偏移,阻碍在线跨模态学习并导致准确率下降。尽管已有工作尝试解决此问题,我们发现当前最优方法存在灾难性遗忘,测试时持续参数更新使性能远低于源模型。本文首次表明,仅对目标域适配模态融合层不仅提升该域表现,还能增强后续域性能。基于融合层参数强跨任务可迁移性,提出无需源数据的$ exttt{AVReCAP}$方法:通过选择性参数检索机制,仅用小批量测试数据从缓冲区动态获取最优融合层参数,融入模型后适应当前分布,并存回以备未来使用。在包含单模态与双模态扰动的基准数据集上实验表明,$ exttt{AVReCAP}$显著优于现有方法,且有效抑制灾难性遗忘。
原文摘要 · Abstract (English)
Audio-visual continual test-time adaptation involves continually adapting a source audio-visual model at test-time, to unlabeled non-stationary domains, where either or both modalities can be distributionally shifted, which hampers online cross-modal learning and eventually leads to poor accuracy. While previous works have tackled this problem, we find that SOTA methods suffer from catastrophic forgetting where the model's performance drops well below even the source model due to continual parameter updates at test-time. In this work, we first show that adapting only the modality fusion layer to a target domain not only improves performance on that domain but can also enhance performance on subsequent domains. Based on this strong cross-task transferability of the fusion layer's parameters, we propose a method, $\texttt{AVReCAP}$, that improves test-time performance of the models without access to any source data. Our approach works by using a selective parameter retrieval mechanism that dynamically retrieves the best fusion layer parameters from a buffer using only a small batch of test data. These parameters are then integrated into the model, adapted to the current test distribution, and saved back for future use. Extensive experiments on benchmark datasets involving unimodal and bimodal corruptions show our proposed $\texttt{AVReCAP}$ significantly outperforms existing methods while minimizing catastrophic forgetting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。