让多模态大模型像人一样推理视觉缺失场景,提升因果解释能力。
AbductiveMLLM: Boosting Visual Abductive Reasoning Within MLLMs
- 用语言和图像双通道模拟人类推理,分别处理假设生成与视觉补全。
- 在标准评测中超越现有模型,显著提升对不完整视觉信息的合理解释能力。
- 适合需要强因果推理的视觉理解任务,如自动驾驶、医疗影像分析。
视觉归纳推理(VAR)是一项挑战性任务,要求人工智能系统对不完整的视觉观察推断最可能的解释。尽管近期多模态大模型(MLLMs)具备强大的通用多模态推理能力,但在归纳推理方面仍远不及人类。为此,我们借鉴人类认知中语言与图像双重归纳的互动机制,提出通过模仿这种双模态行为来增强MLLM的归纳能力。具体而言,我们设计了包含两个协同组件的AbductiveMLLM:REASONER在语言域运作,先用盲目标大模型探索广泛的可能解释,再基于跨模态因果对齐剔除视觉不符的假设;剩余假设作为定向先验输入MLLM,引导其推理趋向因果一致的解释。IMAGINER则通过条件化文本到图像扩散模型,在输入视频和REASONER输出嵌入的基础上“想象”对应的语言解释所对应的合理视觉场景,从而增强上下文感知。两个组件以端到端方式联合训练。在标准VAR基准测试中,AbductiveMLLM实现当前最优性能,持续优于传统方法和先进MLLM。
原文摘要 · Abstract (English)
Visual abductive reasoning (VAR) is a challenging task that requires AI systems to infer the most likely explanation for incomplete visual observations. While recent MLLMs develop strong general-purpose multimodal reasoning capabilities, they fall short in abductive inference, as compared to human beings. To bridge this gap, we draw inspiration from the interplay between verbal and pictorial abduction in human cognition, and propose to strengthen abduction of MLLMs by mimicking such dual-mode behavior. Concretely, we introduce AbductiveMLLM comprising of two synergistic components: REASONER and IMAGINER. The REASONER operates in the verbal domain. It first explores a broad space of possible explanations using a blind LLM and then prunes visually incongruent hypotheses based on cross-modal causal alignment. The remaining hypotheses are introduced into the MLLM as targeted priors, steering its reasoning toward causally coherent explanations. The IMAGINER, on the other hand, further guides MLLMs by emulating human-like pictorial thinking. It conditions a text-to-image diffusion model on both the input video and the REASONER's output embeddings to "imagine" plausible visual scenes that correspond to verbal explanation, thereby enriching MLLMs' contextual grounding. The two components are trained jointly in an end-to-end manner. Experiments on standard VAR benchmarks show that AbductiveMLLM achieves state-of-the-art performance, consistently outperforming traditional solutions and advanced MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。