让大模型看懂图片：用对比解码提升视觉推理能力
Language Models Can See Better: Visual Contrastive Decoding For LLM Multimodal Reasoning
- 通过文本化视觉信息，利用对比解码聚焦新信息
- 在5个数据集上提升准确率，不需额外训练
- 适合想低成本增强视觉理解的开发者
尽管大语言模型（LLMs）在语言任务中表现出色，但其并非为多模态挑战设计。训练多模态大语言模型（MLLMs）成本高昂且受多种限制。本文提出模块化视觉对比解码（MVCD）框架，利用LLMs的上下文学习能力与提出的视觉对比示例解码（CED），无需额外训练即可提升视觉感知。该方法将视觉信号转化为文本，解码时关注对比输出分布，突出上下文示例引入的新信息，探索关联并避免过度依赖已有知识。实验在四个LLMs和五个问答数据集上验证了有效性，结果显示准确率持续提升，并可解释策略内部关键组件的作用。代码将开源于https://github.com/Pbhgit/MVCD。
原文摘要 · Abstract (English)
Although Large Language Models (LLMs) excel in reasoning and generation for language tasks, they are not specifically designed for multimodal challenges. Training Multimodal Large Language Models (MLLMs), however, is resource-intensive and constrained by various training limitations. In this paper, we propose the Modular-based Visual Contrastive Decoding (MVCD) framework to move this obstacle. Our framework leverages LLMs' In-Context Learning (ICL) capability and the proposed visual contrastive-example decoding (CED), specifically tailored for this framework, without requiring any additional training. By converting visual signals into text and focusing on contrastive output distributions during decoding, we can highlight the new information introduced by contextual examples, explore their connections, and avoid over-reliance on prior encoded knowledge. MVCD enhances LLMs' visual perception to make it see and reason over the input visuals. To demonstrate MVCD's effectiveness, we conduct experiments with four LLMs across five question answering datasets. Our results not only show consistent improvement in model accuracy but well explain the effective components inside our decoding strategy. Our code will be available at https://github.com/Pbhgit/MVCD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。