只用问题重播实现持续视觉问答,兼顾记忆与学习。
Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering
- 仅用历史任务问题重播,避免存储图像数据。
- 在VQAv2和NExT-QA上超越现有方法,性能更稳定。
- 适合需要隐私保护和低内存的持续学习场景。
视觉问答中的持续学习(VQACL)要求模型在获得新视觉-语言能力的同时,保留已有知识。由于多模态特性,模型需在视觉与文本域间平衡稳定性并适应新物体和推理任务。现有方法多针对单模态设计,难以满足双重需求。本文提出QUAD:一种仅使用过去任务问题进行正则化的新型方法。通过消除对视觉数据的存储需求,不仅降低内存开销,还缓解隐私问题。该方法引入问题重播机制,选择性复用旧问题以对抗当前任务答案空间的过拟合,解决答案集外的遗忘问题。同时提出注意力一致性蒸馏,强化跨任务的模态内与跨模态注意力一致性,保持关键视觉-语言关联。在VQAv2和NExT-QA上的大量实验表明,QUAD显著优于现有最优方法,在持续视觉问答中表现稳健。代码已开源。
原文摘要 · Abstract (English)
Continual Learning in Visual Question Answering (VQACL) requires models to acquire new visual-linguistic skills (plasticity) while preserving previously learned knowledge (stability). The inherent multimodality of VQACL exacerbates this challenge, as models must balance stability across visual and textual domains while adapting to novel objects and reasoning tasks. Existing methods, primarily designed for unimodal settings, often fall short in addressing this dual requirement. In this work, we present QUestion-only replay with Attention Distillation (QUAD), a novel approach for VQACL that leverages only past task questions for regularization. By eliminating the need to store visual data, QUAD not only reduces memory overhead, but also alleviates privacy concerns. Our method introduces a Question-only Replay mechanism that selectively reuses prior task questions to counteract overfitting to the answer space of the current task, addressing the problem out of answer set. Complementing this, we propose Attention Consistency Distillation to enforce both intra-modal and inter-modal attention consistency across tasks, preserving essential visual-linguistic associations. Extensive experiments on VQAv2 and NExT-QA demonstrate that QUAD significantly outperforms state-of-the-art methods, achieving robust performance in continual VQA. Code is available at: https://github.com/IemProg/QUAD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。