让多模态模型既能快答简单问题,又能深度思考复杂任务。
Metis-HOME: Hybrid Optimized Mixture-of-Experts for Multimodal Reasoning
- 分设思考与非思考两条专家路径,按需分配计算资源。
- 在数学推理等复杂任务上性能提升,同时通用视觉问答能力不降反升。
- 适合追求高效通用多模态推理的开发者和研究者。
受大语言模型推理进展启发,多模态推理领域在数学问题求解等复杂任务上取得显著进步。然而,现有多模态大模型存在两大局限:对简单查询也采用高耗能推理,造成效率低下;过度专注专项推理,损害了其普遍理解能力。本文提出Metis-HOME:一种混合优化的专家混合(MoE)框架,旨在解决这一权衡难题。Metis-HOME通过将原密集模型拆分为两个独立专家分支——面向复杂多步推理的思考分支,以及针对通用VQA和OCR等任务的快速直接推理分支——实现“混合思维”范式。一个轻量级可训练路由机制动态分配查询至最优专家。我们以Qwen2.5-VL-7B为基础构建Metis-HOME MoE架构。全面评估表明,该方法不仅显著增强复杂推理能力,还提升了模型的通用表现,逆转了其他专用推理模型中普遍存在的性能退化趋势。本工作为构建强大且多功能的多模态大模型建立了新范式,有效解决了当前广泛存在的推理与泛化之间的矛盾。代码与权重已开源于https://github.com/MM-Thinking/Metis-HOME。
原文摘要 · Abstract (English)
Inspired by recent advancements in LLM reasoning, the field of multimodal reasoning has seen remarkable progress, achieving significant performance gains on intricate tasks such as mathematical problem-solving. Despite this progress, current multimodal large reasoning models exhibit two key limitations. They tend to employ computationally expensive reasoning even for simple queries, leading to inefficiency. Furthermore, this focus on specialized reasoning often impairs their broader, more general understanding capabilities. In this paper, we propose Metis-HOME: a Hybrid Optimized Mixture-of-Experts framework designed to address this trade-off. Metis-HOME enables a ''Hybrid Thinking'' paradigm by structuring the original dense model into two distinct expert branches: a thinking branch tailored for complex, multi-step reasoning, and a non-thinking branch optimized for rapid, direct inference on tasks like general VQA and OCR. A lightweight, trainable router dynamically allocates queries to the most suitable expert. We instantiate Metis-HOME by adapting the Qwen2.5-VL-7B into an MoE architecture. Comprehensive evaluations reveal that our approach not only substantially enhances complex reasoning abilities but also improves the model's general capabilities, reversing the degradation trend observed in other reasoning-specialized models. Our work establishes a new paradigm for building powerful and versatile MLLMs, effectively resolving the prevalent reasoning-vs-generalization dilemma. Code and weights are available at https://github.com/MM-Thinking/Metis-HOME.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。