arXiv:2506.07778cs.CV2025-06被引 1

用符号逻辑提升视觉推理能力,让AI更懂复杂场景的组合理解。

A Neurosymbolic Agent System for Compositional Visual Reasoning

  • 分两阶段神经符号系统:先生成可解释的推理计划,再执行
  • 引入语法语义检查器,自动修复大模型生成的逻辑错误
  • 关键步骤加入验证机制,提升复杂推理的准确率

大型语言模型(LLM)和视觉模型的发展推动了多模态视觉-语言推理的进步。然而,现有视觉-语言模型在组合式视觉推理方面仍面临挑战。本文提出VLAgent,一种神经符号视觉-语言智能体系统,用于高效实现组合式视觉推理,具有三个创新特性。第一,开发了一种可解释的、增强可视化的两阶段神经符号推理系统:前端引擎利用预训练大模型结合少量示例链式思维上下文学习,为每个组合式视觉推理任务生成结构化推理计划(符号程序脚本);后端引擎将该计划转化为基于视觉输入(图像或视频)的可执行代码,结合神经模型与符号函数,完成一系列动作。第二,为确保逻辑计划到可执行指令映射的质量,引入SS-parser,检查脚本的语法和语义正确性,检测并修复大模型生成的逻辑错误后再生成程序。第三,引入执行验证器,在关键推理步骤中逐步验证和优化结果,例如对高置信度推理采用集成方法,对低置信度推理进行描述分析。在六个视觉基准上的大量实验表明,相比十余个最先进模型,VLAgent在组合式视觉推理上表现更优。

原文摘要 · Abstract (English)

The advancement in large language models (LLMs) and large vision models has fueled the rapid progress in multi-modal vision-language reasoning capabilities. However, existing vision-language models (VLMs) remain challenged by compositional visual reasoning. This paper presents VLAgent, a neuro-symbolic approach to developing a Vision-Language Agent system for efficient compositional visual reasoning with three novel features. First, VLAgent develops an interpretable visualization-enhanced two-stage neuro-symbolic reasoning system. The first stage is managed by a front-end engine that generates a structured visual reasoning plan (symbolic program script) for each compositional visual reasoning task by utilizing a pre-trained LLM powered with few-shot chain-of-thought in-context learning. The second stage is managed by a high-performance back-end engine. It transforms the planning script into executable code based on visual input (image or video) and the combination of neural models and symbolic functions and then performs a sequence of actions for the compositional visual reason task. Second, to ensure and enhance the quality of mapping the logic plan to a sequence of executable instructions, VLAgent introduces the SS-parser, which examines the syntax and semantic correctness of the planning script, detects and repairs the logic errors found in the LLM-generated logic plan before generating the executable program. Third, VLAgent introduces the execution verifier in critical reasoning steps to validate and refine its compositional reasoning results in a stepwise manner, for example, ensemble methods for critical visual reasoning and caption analysis for low-confidence compositional reasoning. Extensive experiments on six visual benchmarks compared to a dozen SoTA visual reasoning models show that VLAgent outperforms existing representative approaches to compositional visual reasoning.

视觉推理神经符号大模型可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。