用可自适应的软提示提升小样本视觉问答能力
Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering
- 通过注意力映射模块从图像特征中提取任务相关软提示
- 仅需少量梯度更新即可在低数据场景下实现性能提升21.2%
- 适合资源受限下快速适配新视觉问答任务的研究者
大型多模态模型（LMMs）常依赖上下文学习（ICL）以极少标注完成新的视觉问答（VQA）任务。然而，小规模LMM在增加示例数时性能未必持续提升，我们推测这是由于图像嵌入中的无关信息干扰了模型。为此，提出一种元学习方法：从任务相关的视觉特征中蒸馏出固定软提示，并在测试时通过少量示例自适应调整。通过注意力映射模块实现蒸馏，该模块可无缝集成至任意LMM架构并与软提示联合训练。在VL-ICL Bench上的评估显示，本方法仅需几步梯度更新即可在低数据场景下实现任务适配，较ICL提升21.2%；相比参数高效微调方法，元学习进一步提升7.7%。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) often rely on in-context learning (ICL) to perform new visual question answering (VQA) tasks with minimal supervision. However, ICL performance, especially in smaller LMMs, does not always improve monotonically when increasing the number of examples. We hypothesize that this happens because the LMM is overwhelmed by extraneous information in the image embeddings that is irrelevant to the downstream task. To address this, we propose a meta-learning approach that induces few-shot capabilities in LMMs through a fixed set of soft prompts distilled from task-relevant visual features, which are adapted at test time using a small number of examples. We facilitate this distillation through an attention-mapper module that can be easily integrated with any LMM architecture and is jointly learned with soft prompts. Evaluation on the VL-ICL Bench shows that our method successfully achieves task adaptation in low-data regimes with just a few gradient steps, outperforming ICL by 21.2%. Comparisons with parameter-efficient finetuning methods demonstrate that meta-learning further enhances this adaptation by 7.7% for various VQA tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。