用神经符号树增强视觉语言模型的组合推理能力
COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Models
- 基于LLM构建可解释的概念树，提升视觉语言模型的语义理解
- 在四个基准测试中，组合泛化性能提升5-10%
- 适用于需要可解释推理的多模态应用，如医疗图像分析
组合推理仍是现代视觉语言模型（VLMs）的持续弱点：当任务依赖于理解图像中多个物体、属性和关系的交互时，模型常表现不佳。现有方法通过改进提示结构、链式思维等技巧尝试提升，近期研究则利用训练良好的大语言模型（LLMs）来增强VLM的推理能力，以弥补其语言理解短板。但这些方法往往资源消耗大或缺乏可解释性。本文提出COCO-Tree——一种从LLM学习得到的精心设计的神经符号概念树，用于增强VLM输出，提升其语言推理能力。该方法采用类似束搜索的推理过程，在四个组合性基准（Winoground、EqBench、ColorSwap、SugarCrepe）上，对七种不同规模的开源VLM进行评估，结果显示，与基线相比，组合泛化性能显著提升5%-10%。
原文摘要 · Abstract (English)
Compositional reasoning remains a persistent weakness of modern vision language models (VLMs): they often falter when a task hinges on understanding how multiple objects, attributes, and relations interact within an image. Multiple research works have attempted to improve compositionality performance by creative tricks such as improving prompt structure, chain of thought reasoning, etc. A more recent line of work attempts to impart additional reasoning in VLMs using well-trained Large Language Models (LLMs), which are far superior in linguistic understanding than VLMs to compensate for the limited linguistic prowess of VLMs. However, these approaches are either resource-intensive or do not provide an interpretable reasoning process. In this paper, we present 'COCO-Tree' - a novel approach that augments VLM outputs with carefully designed neurosymbolic concept trees learned from LLMs to improve VLM's linguistic reasoning. COCO-Tree's beam search-inspired reasoning process boosts compositionality performance and provides a rationale behind VLM predictions. Empirical results on four compositionality benchmarks, Winoground, EqBench, ColorSwap, and SugarCrepe, in seven different open-source VLMs with varying sizes, demonstrate that COCO-Tree significantly improves compositional generalization by 5-10% over baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。