Corvid提升多模态模型链式推理能力,解决复杂问题时的思维断裂问题。
Corvid: Improving Multimodal Large Language Models Towards Chain-of-Thought Reasoning
- 采用混合视觉编码器与门控连接器,强化图文对齐与信息表达
- 用28.7万条高质量多模态推理数据微调,实现分步推理能力跃升
- 引入推理时自验证机制,防止过度或不足推理,适合科学与数学任务
近期多模态大语言模型(MLLMs)在多模态感知与理解方面表现优异,但主流开源模型在需要深度推理的复杂任务中仍存在明显短板。本文提出Corvid,一种具备增强链式思维(CoT)能力的MLLM。其架构包含混合视觉编码器以生成丰富视觉表征,并设计了门控连接器(GateMixer)促进跨模态对齐。为提升CoT推理能力,我们构建了MCoT-Instruct-287K数据集,该数据集由多种公开推理源经精炼与标准化而来。通过两阶段CoT格式训练策略,逐步增强模型分步推理能力。此外,提出一种高效的推理时缩放策略,使Corvid能通过自验证缓解过度推理与推理不足问题。大量实验表明,Corvid超越现有类似o1的MLLM及同等参数量的先进模型,在数学推理与科学问题求解上表现突出。
原文摘要 · Abstract (English)
Recent advancements in multimodal large language models (MLLMs) have demonstrated exceptional performance in multimodal perception and understanding. However, leading open-source MLLMs exhibit significant limitations in complex and structured reasoning, particularly in tasks requiring deep reasoning for decision-making and problem-solving. In this work, we present Corvid, an MLLM with enhanced chain-of-thought (CoT) reasoning capabilities. Architecturally, Corvid incorporates a hybrid vision encoder for informative visual representation and a meticulously designed connector (GateMixer) to facilitate cross-modal alignment. To enhance Corvid's CoT reasoning capabilities, we introduce MCoT-Instruct-287K, a high-quality multimodal CoT instruction-following dataset, refined and standardized from diverse public reasoning sources. Leveraging this dataset, we fine-tune Corvid with a two-stage CoT-formatted training approach to progressively enhance its step-by-step reasoning abilities. Furthermore, we propose an effective inference-time scaling strategy that enables Corvid to mitigate over-reasoning and under-reasoning through self-verification. Extensive experiments demonstrate that Corvid outperforms existing o1-like MLLMs and state-of-the-art MLLMs with similar parameter scales, with notable strengths in mathematical reasoning and science problem-solving. Project page: https://mm-vl.github.io/corvid.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。