用视觉语言模型自动发现规则,结合逻辑网络验证修正,实现高效视觉推理。
Think-Verify-Revise: Neuro-Symbolic Visual Reasoning with Vision-Language Models and Dynamic Logic Tensor Networks

- 通过视觉语言模型生成候选逻辑规则,动态构建逻辑网络进行验证。
- 仅用3个训练样本即可在4个数据集上正确推导数独约束规则,性能媲美或超越基线。
- 适合需要自动规则学习的视觉推理场景,尤其适用于小样本复杂关系建模。
视觉推理任务要求系统同时感知视觉内容并应用形式化关系约束,而纯神经或纯符号方法难以单独胜任。本文提出一种神经符号框架,通过将视觉语言模型(VLM)与动态逻辑张量网络(D-LTN)紧密耦合,在闭环迭代反馈中实现规则发现与验证。VLM接收少量标注视觉样例,生成符合严格语法的的一阶逻辑(FOL)规则(思考);D-LTN实时由这些规则构建,基于CNN提取的视觉嵌入对规则进行可微验证(验证);验证失败结果反向引导VLM生成新假设(修正)。在ViSudo-PC基准测试中,该系统仅用三个训练样本即成功推导出数独约束规则,覆盖MNIST、EMNIST、KMNIST和FMNIST四个视觉领域,其AUC表现达到或超过先前方法(NeuPSL、LTN),证明了利用视觉语言模型实现自动规则发现的潜力。代码已开源:https://github.com/homayoun-afshari/nesy。
原文摘要 · Abstract (English)
Visual reasoning tasks require a system to jointly perceive visual content and apply formal relational constraints---a combination that neither pure neural nor purely symbolic approaches handle well in isolation. This paper proposes a Neuro-Symbolic (NeSy) framework that closes this gap by tightly coupling a Vision-Language Model (VLM) for automatic First-Order Logic (FOL) rule induction with a Dynamic Logic Tensor Network (D-LTN) for differentiable rule verification, in a closed iterative feedback loop. The VLM receives a small set of labelled visual examples and proposes candidate FOL rules conforming to a strict grammar (Think); the D-LTN is automatically assembled from these rules at runtime and evaluates them grounding on CNN-produced visual embeddings (Verify); and verification failures are fed back to guide the VLM's next hypothesis (Revise). Evaluated on the ViSudo-PC benchmark across four visual domains (MNIST, EMNIST, KMNIST, FMNIST), the system induces valid Sudoku constraint rules using only three training examples as visual context. The proposed method achieves AUC scores matching or outperforming previous methods (NeuPSL, LTN), showing the potential for automatic rule discovery through VLM. Code is available at https://github.com/homayoun-afshari/nesy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。