用混合偏好优化提升多模态大模型的推理能力，显著改进思维链表现。
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
- 构建自动化数据管道生成高质量多模态推理偏好数据集MMPR。
- 提出MPO方法，在InternVL2-8B上使MathVista准确率提升至67.0，超原版8.7点。
- 仅用小模型实现接近十倍大模型的推理性能，适合追求效率的开发者。
现有开源多模态大模型通常采用预训练与监督微调流程，但存在分布偏移问题，限制了其多模态推理能力，尤其在思维链（CoT）任务中表现不佳。为此，我们引入偏好优化（PO）流程以增强多模态大模型的推理能力。具体而言：(1) 在数据层面，设计自动化偏好数据构建流水线，创建高质量、大规模的多模态推理偏好数据集MMPR；(2) 在模型层面，探索将PO与多模态大模型结合，提出一种简单有效的混合偏好优化（MPO）方法，显著提升多模态思维链性能。该方法有效增强了InternVL2-8B和InternVL2-76B的推理能力。特别地，我们的模型InternVL2-8B-MPO在MathVista上达到67.0的准确率，比原版高出8.7点，性能接近10倍大的InternVL2-76B。代码、数据及模型均已公开。
原文摘要 · Abstract (English)
Existing open-source multimodal large language models (MLLMs) generally follow a training process involving pre-training and supervised fine-tuning. However, these models suffer from distribution shifts, which limit their multimodal reasoning, particularly in the Chain-of-Thought (CoT) performance. To address this, we introduce a preference optimization (PO) process to enhance the multimodal reasoning capabilities of MLLMs. Specifically, (1) on the data side, we design an automated preference data construction pipeline to create MMPR, a high-quality, large-scale multimodal reasoning preference dataset; and (2) on the model side, we explore integrating PO with MLLMs, developing a simple yet effective method, termed Mixed Preference Optimization (MPO), which boosts multimodal CoT performance. Our approach enhances the multimodal reasoning abilities of both InternVL2-8B and InternVL2-76B. Notably, our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance comparable to the 10$\times$ larger InternVL2-76B. We hope this study could inspire further advancements in MLLMs. Code, data, and model are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。