首个多模态符号逻辑推理基准，揭示大模型在严谨推理上的严重短板。
MuSLR: Multimodal Symbolic Logical Reasoning
- 构建基于形式逻辑的多模态推理框架，用规则约束跨模态理解
- 7个主流模型在新基准上最高仅达46.8%，暴露逻辑对齐缺陷
- 提出LogiCAM模块化框架，显著提升复杂逻辑推理准确率
多模态符号逻辑推理旨在通过形式逻辑从多模态输入中推导新事实，对自动驾驶、医疗诊断等高风险场景至关重要。为评估当前视觉语言模型（VLMs）在此类任务上的能力，我们引入首个基于形式逻辑规则的多模态符号逻辑推理基准MuSLR。MuSLR包含1,093个实例，覆盖7个领域，含35个原子符号逻辑和976个逻辑组合，推理深度为2至9。我们在7个先进VLM上测试，发现所有模型均表现不佳，最佳模型GPT-4.1仅达46.8%。为此，我们提出LogiCAM模块化框架，将形式逻辑规则应用于多模态输入，使GPT-4.1的链式思考性能提升14.13%，在复杂逻辑如一阶逻辑上增益更大。我们还进行了全面错误分析，发现约70%失败源于模态间逻辑不一致，为未来改进提供关键方向。所有数据与代码已公开于https://llm-symbol.github.io/MuSLR。
原文摘要 · Abstract (English)
Multimodal symbolic logical reasoning, which aims to deduce new facts from multimodal input via formal logic, is critical in high-stakes applications such as autonomous driving and medical diagnosis, as its rigorous, deterministic reasoning helps prevent serious consequences. To evaluate such capabilities of current state-of-the-art vision language models (VLMs), we introduce the first benchmark MuSLR for multimodal symbolic logical reasoning grounded in formal logical rules. MuSLR comprises 1,093 instances across 7 domains, including 35 atomic symbolic logic and 976 logical combinations, with reasoning depths ranging from 2 to 9. We evaluate 7 state-of-the-art VLMs on MuSLR and find that they all struggle with multimodal symbolic reasoning, with the best model, GPT-4.1, achieving only 46.8%. Thus, we propose LogiCAM, a modular framework that applies formal logical rules to multimodal inputs, boosting GPT-4.1's Chain-of-Thought performance by 14.13%, and delivering even larger gains on complex logics such as first-order logic. We also conduct a comprehensive error analysis, showing that around 70% of failures stem from logical misalignment between modalities, offering key insights to guide future improvements. All data and code are publicly available at https://llm-symbol.github.io/MuSLR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。