提出轻量模块化框架CATCH,让VQA模型跨领域通用无需重训练。
CATCH: A Modular Cross-domain Adaptive Template with Hook
- 用分类器+双适配器动态注入视觉与语言调整模块。
- 在4个跨域数据集上提升性能,最高增3.1分。
- 适合需要快速部署到新领域的VQA系统使用。
近期视觉问答(VQA)模型在自然图像上表现优异,如基于大语言模型的LLaVA能进行开放式推理。然而在遥感、医学影像或数学图表等域外场景中,因分布差异大且缺乏有效适应机制,泛化能力显著下降。现有方法多依赖各领域微调或定制流程,成本高、不灵活且难扩展。本文提出CATCH,一种即插即用的跨域自适应框架,在不改变主干模型的前提下提升VQA泛化能力。核心思想是将视觉与语言适应解耦:引入域分类器识别图像类型,并设计双适配器机制——提示适配器调节语言,视觉适配器调整视觉特征。两者通过统一钩子接口动态注入,无需重训练主干。在四个特定领域VQA基准测试中,该框架实现一致性能提升,包括MathVQA上BLEU提升+2.3,MedVQA-RAD上VQA得分+2.6,ChartQA上ROUGE提升+3.1。结果表明,CATCH为多领域VQA提供了可扩展、易拓展的解决方案,支持在多样化应用场景中的实用部署。
原文摘要 · Abstract (English)
Recent advances in Visual Question Answering (VQA) have demonstrated impressive performance in natural image domains, with models like LLaVA leveraging large language models (LLMs) for open-ended reasoning. However, their generalization degrades significantly when transferred to out-of-domain scenarios such as remote sensing, medical imaging, or math diagrams, due to large distributional shifts and the lack of effective domain adaptation mechanisms. Existing approaches typically rely on per-domain fine-tuning or bespoke pipelines, which are costly, inflexible, and not scalable across diverse tasks. In this paper, we propose CATCH, a plug-and-play framework for cross-domain adaptation that improves the generalization of VQA models while requiring minimal changes to their core architecture. Our key idea is to decouple visual and linguistic adaptation by introducing two lightweight modules: a domain classifier to identify the input image type, and a dual adapter mechanism comprising a Prompt Adapter for language modulation and a Visual Adapter for vision feature adjustment. Both modules are dynamically injected via a unified hook interface, requiring no retraining of the backbone model. Experimental results across four domain-specific VQA benchmarks demonstrate that our framework achieves consistent performance gains without retraining the backbone model, including +2.3 BLEU on MathVQA, +2.6 VQA on MedVQA-RAD, and +3.1 ROUGE on ChartQA. These results highlight that CATCH provides a scalable and extensible approach to multi-domain VQA, enabling practical deployment across diverse application domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。