构建首个可程序验证的多模态深度复合推理基准,测试模型在复杂视觉任务中的逻辑推理能力。
MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoning
- 设计分层推理链,每层包含多对象、属性与关系组合的视觉条件判断。
- 最强模型路径F1仅53.33,复杂度提升时性能显著下降。
- 适用于评估大模型在GUI导航、图表理解等真实场景中的推理能力。
多模态大语言模型(MLLMs)正被用于执行如GUI导航等视觉工作流,其中下一步取决于经验证的视觉复合条件(如“若出现权限对话框且界面为绿色,则点击允许”),流程可能分支或提前终止。然而,该能力仍缺乏有效评估:现有基准多聚焦浅层组合或独立约束,而非深层嵌套的复合条件。本文提出MM-CondChain,一个面向视觉引导的深度复合推理基准。每个实例由多层推理链构成,每层包含基于视觉证据的非平凡复合条件,由多个对象、属性或关系构建而成。正确回答需模型在细节上感知图像,在每步推理多个视觉元素,并沿生成路径到达最终结果。为可扩展构造此类工作流数据,我们提出代理式合成流水线:规划器逐层生成复合条件,可验证的程序化中间表示(VPIR)确保每层条件机械可验证,组装器将这些已验证层整合为完整指令。利用此流水线,我们在自然图像、数据图表和GUI轨迹三个视觉领域构建了基准。对多种MLLM的实验表明,即使最强模型路径F1也仅为53.33,且在困难负例、深度或谓词复杂度增加时性能急剧下降,证实深度复合推理仍是根本挑战。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) are increasingly used to carry out visual workflows such as navigating GUIs, where the next step depends on verified visual compositional conditions (e.g., "if a permission dialog appears and the color of the interface is green, click Allow") and the process may branch or terminate early. Yet this capability remains under-evaluated: existing benchmarks focus on shallow-compositions or independent-constraints rather than deeply chained compositional conditionals. In this paper, we introduce MM-CondChain, a benchmark for visually grounded deep compositional reasoning. Each benchmark instance is organized as a multi-layer reasoning chain, where every layer contains a non-trivial compositional condition grounded in visual evidence and built from multiple objects, attributes, or relations. To answer correctly, an MLLM must perceive the image in detail, reason over multiple visual elements at each step, and follow the resulting execution path to the final outcome. To scalably construct such workflow-style data, we propose an agentic synthesis pipeline: a Planner orchestrates layer-by-layer generation of compositional conditions, while a Verifiable Programmatic Intermediate Representation (VPIR) ensures each layer's condition is mechanically verifiable. A Composer then assembles these verified layers into complete instructions. Using this pipeline, we construct benchmarks across three visual domains: natural images, data charts, and GUI trajectories. Experiments on a range of MLLMs show that even the strongest model attains only 53.33 Path F1, with sharp drops on hard negatives and as depth or predicate complexity grows, confirming that deep compositional reasoning remains a fundamental challenge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。