无需训练即可让多模态大模型具备深度推理与反思能力
Training-Free Reasoning and Reflection in MLLMs
- 通过分层融合视觉预训练模型与推理专用语言模型，实现无梯度更新的推理增强
- 在MMMU基准上达到69.2准确率，超越InternVL2.5-38B和GPT-4o
- 适合希望低成本提升多模态模型推理能力的研究者与开发者
近期的推理型大语言模型（如DeepSeek-R1和OpenAI-o1）通过强化学习展现出强大推理能力。然而，将此类能力扩展至多模态大模型（MLLMs）受限于重训练成本高昂及高质量可验证多模态推理数据集稀缺。本文提出FRANK Model——一种无需训练、类r1的MLLM，可在不进行梯度更新或额外监督的情况下赋予现成MLLM推理与反思能力。核心洞察是：相较于深层解码器层，浅层解码器层更关注视觉标记，而深层则聚焦文本语义。基于此，我们提出分层权重融合方法，将视觉预训练MLLM与推理专用LLM结合。进一步设计基于泰勒展开的闭式融合机制，将推理能力注入深层解码器，同时保持浅层对视觉信息的精准锚定。大量实验表明该方法有效：在挑战性多模态推理基准MMMU上，FRANK-38B模型达69.2准确率，较最强基线InternVL2.5-38B提升+5.3，甚至超过专有模型GPT-4o。
原文摘要 · Abstract (English)
Recent advances in Reasoning LLMs (e.g., DeepSeek-R1 and OpenAI-o1) have showcased impressive reasoning capabilities via reinforcement learning. However, extending these capabilities to Multimodal LLMs (MLLMs) is hampered by the prohibitive costs of retraining and the scarcity of high-quality, verifiable multimodal reasoning datasets. This paper introduces FRANK Model, a training-FRee ANd r1-liKe MLLM that imbues off-the-shelf MLLMs with reasoning and reflection abilities, without any gradient updates or extra supervision. Our key insight is to decouple perception and reasoning across MLLM decoder layers. Specifically, we observe that compared to the deeper decoder layers, the shallow decoder layers allocate more attention to visual tokens, while the deeper decoder layers concentrate on textual semantics. This observation motivates a hierarchical weight merging approach that combines a visual-pretrained MLLM with a reasoning-specialized LLM. To this end, we propose a layer-wise, Taylor-derived closed-form fusion mechanism that integrates reasoning capacity into deep decoder layers while preserving visual grounding in shallow decoder layers. Extensive experiments on challenging multimodal reasoning benchmarks demonstrate the effectiveness of our approach. On the MMMU benchmark, our model FRANK-38B achieves an accuracy of 69.2, outperforming the strongest baseline InternVL2.5-38B by +5.3, and even surpasses the proprietary GPT-4o model. Our project homepage is at: http://iip.whu.edu.cn/frank/index.html
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。