测试时动态优化多模态模型，提升复杂推理能力
Test-Time Warmup for Multimodal Large Language Models
- 测试时通过弱监督任务数据微调模型，实现个性化适应
- 在MMMU、VQA-Rad等数据集上分别提升4.03%、5.28%
- 适合需要高鲁棒性推理的多模态应用开发者
多模态大语言模型（MLLMs）在文本与图像交叉推理方面潜力巨大，但尚未充分实现。这类模型通常由预训练的语言模型、视觉编码器及连接两者嵌入空间的模块组成，尽管各组件均在数十亿样本上预训练，但整体模型仅在数千至数百万样本上微调，导致复杂推理性能不足。为此，我们提出一种测试时暖化（Test-Time Warmup）方法，不依赖大量标注数据，而是利用弱监督辅助任务的数据，在每个测试实例上动态调整模型。实验显示，该方法使Llama-Vision-Instruct模型在MMMU上相对提升4.03%，在VQA-Rad上提升5.28%，在GQA上提升1.63%。结果表明，推理前的‘暖化’操作能显著增强模型在多样化推理任务中的鲁棒性。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) hold great promise for advanced reasoning at the intersection of text and images, yet they have not fully realized this potential. MLLMs typically integrate an LLM, a vision encoder, and a connector that maps the vision encoder's embeddings into the LLM's text embedding space. Although each component is pretrained on massive datasets with billions of samples, the entire multimodal model is typically trained on only thousands (or a few million) samples, which can result in weak performance on complex reasoning tasks. To address these shortcomings, instead of relying on extensive labeled datasets for fine-tuning, we propose a Test-Time Warmup method that adapts the MLLM per test instance by leveraging data from weakly supervised auxiliary tasks. With our approach, we observe a relative performance improvement of 4.03% on MMMU, 5.28% on VQA-Rad, and 1.63% on GQA on the Llama-Vision-Instruct model. Our method demonstrates that 'warming up' before inference can enhance MLLMs' robustness across diverse reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。