动态生成低秩适配器，让模型持续学习新问题时不忘旧知识。
HyLoVQA: Dynamic Hypernetwork-Generated Low-Rank Adaptation for Continual Visual Question Answering
- 用锚点记忆库存储视觉与语言特征，动态检索并生成轻量适配器。
- 在VQA v2和NExT-QA上性能超越当前最佳方法，保持长期记忆不遗忘。
- 适合需要持续学习的视觉问答场景，尤其关注参数效率与任务专注性。
持续视觉问答（Continual VQA）需从非平稳的视觉输入与问题流中学习，同时保留过往知识。现有方法多通过更新大量共享参数实现适应，易引发跨任务干扰，影响对当前任务与对象的准确理解。为此，我们提出HyLoVQA：维护一个抗漂移的锚点记忆库，存储视觉对象与文本任务的内容，并基于当前输入特征动态更新。在检索到的锚点条件下，由超网络生成轻量级低秩适配器（LoRA），实现高效参数调整，使模型能动态适应每个任务与对象。此外，设计对齐损失，约束参数空间中的功能变化与特征空间中的语义差异一致，确保适配器聚焦于当前任务与对象。在VQA v2与NExT-QA数据集上，标准与组合设置下的实验均表明，HyLoVQA显著优于现有最优方法。
原文摘要 · Abstract (English)
Continual Visual Question Answering (VQA) requires learning from non-stationary streams of visual inputs and questions while preserving past knowledge. Most prior methods adapt by updating a largely shared parameter set. This often leads to cross-level task interference, hindering accurate adaptation to the current task and object. To address this limitation, we propose HyLoVQA. It maintains a drift-resilient memory bank of anchors. The bank stores the content of visual objects and textual tasks, and they are updated using current input features. Conditioned on retrieved anchors, a hypernetwork generates lightweight Low-Rank Adaptation (LoRA) adapters. This ensures parameter efficiency, allowing the model to adapt to each task and object dynamically. Additionally, we formulate an alignment loss that aligns semantic discrepancies in the feature space with functional changes in the parameter space, thereby constraining LoRA adapters to remain focused on the current task and object. Extensive experiments on VQA v2 and NExT-QA under both standard and compositional settings demonstrate the superiority of HyLoVQA over prior state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。