arXiv:2411.02210cs.CV2024-11被引 7

用视觉语言模型自动生成数据,解决持续VQA中的遗忘问题。

One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answering

  • 利用VLM生成过往任务的伪重演数据,无需存储真实历史数据。
  • 在两个基准上表现优于所有无数据基线方法,接近有历史数据的方法。
  • 通过统计或聚类平衡生成数据分布,避免高频问题偏差。

视觉语言模型(VLM)在利用大规模多模态数据集方面在视觉问答(VQA)任务中展现出巨大潜力。然而,这些模型在持续学习过程中容易因灾难性遗忘而表现下降。为缓解此问题,重演策略通过在学习新任务时使用旧任务数据来提升性能。但该策略需要存储历史数据,可能受硬件限制或隐私问题制约。本文提出首个无需数据的方法——GaB,利用VLM的语言生成能力,在新任务数据上生成过去任务的问题,从而构建伪重演数据。然而,由于训练数据有限且任务特定,生成问题的分布会偏向高频问题。为此,我们引入伪重演平衡模块,通过问题元统计信息或无监督聚类方法,使生成数据分布更贴近真实数据分布。我们在两个最新基准(VQACL-VQAv2 和 CLOVE-function)上评估了该方法,结果表明,GaB显著优于所有无数据基线方法,在持续学习中保持了更高的VQA性能,且与拥有历史数据的方法相当。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have shown significant promise in Visual Question Answering (VQA) tasks by leveraging web-scale multimodal datasets. However, these models often struggle with continual learning due to catastrophic forgetting when adapting to new tasks. As an effective remedy to mitigate catastrophic forgetting, rehearsal strategy uses the data of past tasks upon learning new task. However, such strategy incurs the need of storing past data, which might not be feasible due to hardware constraints or privacy concerns. In this work, we propose the first data-free method that leverages the language generation capability of a VLM, instead of relying on external models, to produce pseudo-rehearsal data for addressing continual VQA. Our proposal, named as GaB, generates pseudo-rehearsal data by posing previous task questions on new task data. Yet, despite being effective, the distribution of generated questions skews towards the most frequently posed questions due to the limited and task-specific training data. To mitigate this issue, we introduce a pseudo-rehearsal balancing module that aligns the generated data towards the ground-truth data distribution using either the question meta-statistics or an unsupervised clustering method. We evaluate our proposed method on two recent benchmarks, \ie VQACL-VQAv2 and CLOVE-function benchmarks. GaB outperforms all the data-free baselines with substantial improvement in maintaining VQA performance across evolving tasks, while being on-par with methods with access to the past data.

视觉问答持续学习生成数据自回归

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。