让多模态模型根据问题自动切换思考模式,提升视觉任务表现。
SwimBird: Eliciting Switchable Reasoning Mode in Hybrid Autoregressive MLLMs
- 通过动态切换文本、视觉或混合推理模式,适应不同任务需求。
- 在92K条指令数据上训练,实现三种模式的精准识别与切换。
- 兼顾逻辑推理与视觉理解,适合复杂多模态任务场景。
多模态大语言模型(MLLM)在融合视觉与语言方面取得显著进展,但多数模型仅依赖文本思维链(CoT)进行推理,限制了其在视觉密集型任务中的表现。现有方法引入固定数量的连续隐藏状态作为‘视觉思维’,虽提升了视觉性能,却常损害文本逻辑推理能力。本文指出核心问题是预设的僵化推理模式无法根据用户查询自适应选择最优思维模态。为此提出SwimBird,一种可切换推理模式的混合自回归MLLM,能根据输入动态选择:(1) 仅文本推理,(2) 仅视觉推理(以连续隐藏状态表示视觉思维),(3) 视觉-文本交替推理。采用统一的混合自回归框架,同时支持文本思想的下一项预测与视觉思想的下一嵌入预测,并设计系统化的推理模式校准策略,构建了涵盖三种推理模式的监督微调数据集SwimBird-SFT-92K。实验表明,该模型在多种基准测试中均达到最先进水平,相比固定模式方法表现出显著且稳健的提升。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have made remarkable progress in multimodal perception and reasoning by bridging vision and language. However, most existing MLLMs perform reasoning primarily with textual CoT, which limits their effectiveness on vision-intensive tasks. Recent approaches inject a fixed number of continuous hidden states as "visual thoughts" into the reasoning process and improve visual performance, but often at the cost of degraded text-based logical reasoning. We argue that the core limitation lies in a rigid, pre-defined reasoning pattern that cannot adaptively choose the most suitable thinking modality for different user queries. We introduce SwimBird, a reasoning-switchable MLLM that dynamically switches among three reasoning modes conditioned on the input: (1) text-only reasoning, (2) vision-only reasoning (continuous hidden states as visual thoughts), and (3) interleaved vision-text reasoning. To enable this capability, we adopt a hybrid autoregressive formulation that unifies next-token prediction for textual thoughts with next-embedding prediction for visual thoughts, and design a systematic reasoning-mode curation strategy to construct SwimBird-SFT-92K, a diverse supervised fine-tuning dataset covering all three reasoning patterns. By enabling flexible, query-adaptive mode selection, SwimBird preserves strong textual logic while substantially improving performance on vision-dense tasks. Experiments across diverse benchmarks covering textual reasoning and challenging visual understanding demonstrate that SwimBird achieves state-of-the-art results and robust gains over prior fixed-pattern multimodal reasoning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。