用少量参数微调Q-Former，实现与全量微调相当的视觉推理性能。
Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks
- 仅用2%参数通过PEFT微调Q-Former，效果接近全量微调。
- 在ScienceQA和IconQA上，参数效率提升超98%。
- 自注意力层对感知类任务更关键，前馈层重要性随任务复杂度变化。
大型语言模型在视觉推理任务中通过额外编码器实现多模态对齐。尽管Q-Former被广泛用于图像、视频、音频和3D数据与大语言模型的对齐，但其高效训练及组件分析研究仍有限。本文基于InstructBLIP，在ScienceQA和IconQA等视觉推理基准上，探索了参数高效微调（PEFT）对Q-Former的效果。结果表明，使用PEFT仅需不到2%的可训练参数即可达到全量微调的性能。此外，采用AdaLoRA动态分配参数预算，分析了不同子层在4个基准上的相对重要性：自注意力层在感知类任务中更为关键；前馈网络（FFN）层的重要性取决于任务中涉及的视觉-语言模式复杂度。代码已开源：https://github.com/AttentionX/InstructBLIP_PEFT。
原文摘要 · Abstract (English)
Recent advancements in large language models have demonstrated enhanced capabilities in visual reasoning tasks by employing additional encoders for aligning different modalities. While the Q-Former has been widely used as a general encoder for aligning several modalities including image, video, audio, and 3D with large language models, previous works on its efficient training and the analysis of its individual components have been limited. In this work, we investigate the effectiveness of parameter efficient fine-tuning (PEFT) the Q-Former using InstructBLIP with visual reasoning benchmarks ScienceQA and IconQA. We observe that applying PEFT to the Q-Former achieves comparable performance to full fine-tuning using under 2% of the trainable parameters. Additionally, we employ AdaLoRA for dynamic parameter budget reallocation to examine the relative importance of the Q-Former's sublayers with 4 different benchmarks. Our findings reveal that the self-attention layers are noticeably more important in perceptual visual-language reasoning tasks, and relative importance of FFN layers depends on the complexity of visual-language patterns involved in tasks. The code is available at https://github.com/AttentionX/InstructBLIP_PEFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。