提出新框架,让AI像人一样看多张图并思考
CMMCoT: Enhancing Complex Multi-Image Comprehension via Multi-Modal Chain-of-Thought and Memory Augmentation
- 用视觉区域令牌引导跨图像推理,提升理解能力
- 推理时动态记忆关键信息,增强分析深度
- 适合需要多图逻辑推断的场景,如医学影像分析
现有多模态慢思考方法在单图理解中表现优异,但在复杂多图任务中效果受限,主要因过度依赖文本推理。人类在处理多图时会进行持续的跨图视觉比对和动态记忆关键视觉概念。为此,我们提出复杂多模态思维链(CMMCoT)框架,模拟人类“慢思考”过程。该框架包含两项创新:(1) 构建交错的多模态多步推理链,利用中间步骤提取的关键视觉区域令牌作为监督信号,促进跨模态理解并提升可解释性;(2) 引入测试时记忆增强模块,在不增加参数的前提下扩展推理能力。此外,我们构建了一个新的多图慢思考数据集。大量实验验证了模型有效性。代码已开源。
原文摘要 · Abstract (English)
While previous multimodal slow-thinking methods have demonstrated remarkable success in single-image understanding scenarios, their effectiveness becomes fundamentally constrained when extended to more complex multi-image comprehension tasks. This limitation stems from their predominant reliance on text-based intermediate reasoning processes. While for human, when engaging in sophisticated multi-image analysis, they typically perform two complementary cognitive operations: (1) continuous cross-image visual comparison through region-of-interest matching, and (2) dynamic memorization of critical visual concepts throughout the reasoning chain. Motivated by these observations, we propose the Complex Multi-Modal Chain-of-Thought (CMMCoT) framework, a multi-step reasoning framework that mimics human-like "slow thinking" for multi-image understanding. Our approach incorporates two key innovations: (1) The construction of interleaved multimodal multi-step reasoning chains, which utilize critical visual region tokens, extracted from intermediate reasoning steps, as supervisory signals. This mechanism not only facilitates comprehensive cross-modal understanding but also enhances model interpretability. (2) The introduction of a test-time memory augmentation module that expands the model's reasoning capacity during inference while preserving parameter efficiency. Furthermore, to facilitate research in this direction, we have curated a novel multi-image slow-thinking dataset. Extensive experiments demonstrate the effectiveness of our model. Code is available at https://github.com/zhangguanghao523/CMMCoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。