让视觉语言模型像写代码一样灵活推理，解决组合问题。
NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language
- 将自然语言转为可执行的Python程序，融合感知与符号推理
- 无需训练，在多个基准上显著超越基线模型
- 适合需要灵活组合推理的研究者和开发者
现代视觉语言模型在诸多任务中表现优异，但在组合推理方面仍存不足，即难以分解并重组概念以应对新问题。现有神经符号方法常受限于严格逻辑或预定义谓词，灵活性不足。本文提出NePTune，一种神经-符号混合框架，通过融合基础视觉模型的感知能力与符号推理的组合表达力，实现动态将自然语言查询转化为包含可微操作的执行程序。该程序结合了命令式控制流与软逻辑算子，可处理视觉语言模型产生的不确定性。NePTune采用模块化设计，感知与推理解耦，同时支持微调。在多个视觉推理基准及不同领域上进行评估，包括对抗性测试，结果表明其显著优于强基线模型，并展现出出色的组合泛化与新环境适应能力。
原文摘要 · Abstract (English)
Modern Vision-Language Models (VLMs) have achieved impressive performance in various tasks, yet they often struggle with compositional reasoning, the ability to decompose and recombine concepts to solve novel problems. While neuro-symbolic approaches offer a promising direction, they are typically constrained by crisp logical execution or predefined predicates, which limit flexibility. In this work, we introduce NePTune, a neuro-symbolic framework that overcomes these limitations through a hybrid execution model that integrates the perception capabilities of foundation vision models with the compositional expressiveness of symbolic reasoning. NePTune dynamically translates natural language queries into executable Python programs that blend imperative control flow with soft logic operators capable of reasoning over VLM-generated uncertainty. Operating in a training-free manner, NePTune, with a modular design, decouples perception from reasoning, yet its differentiable operations support fine-tuning. We evaluate NePTune on multiple visual reasoning benchmarks and various domains, utilizing adversarial tests, and demonstrate a significant improvement over strong base models, as well as its effective compositional generalization and adaptation capabilities in novel environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。