arXiv:2508.04453cs.CV2025-08IJCAI被引 2

通过因果推理补全图像物体,提升视觉语言模型感知能力。

Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completion

  • 基于因果关系推理被遮挡物体,构建新训练任务CVC。
  • 在4个专项任务上提升5.4%(LLaVA-7B)和4.0%(LLaVA-13B)。
  • 无需GPT-4V或人工标注,可自动构造训练数据自进化。

近年来,大型视觉语言模型(LVLMs)取得了显著进展,但在需要深度视觉感知的任务(如识别图像间细微差异)中表现仍不足。这可能源于主流指令微调语料库中视觉知识的匮乏,导致视觉感知与推理能力欠缺。为此,我们提出一种基于新任务——因果驱动的视觉物体补全(CVC)的自改进框架。该任务要求模型根据可见信息与其之间的因果关系推断被遮挡物体。我们通过自动化实例构建流程低成本获取丰富样本,不依赖复杂模型(如GPT-4V)或人工干预。随后,LVLMs利用这些生成样本进行试错学习,实现有效自提升。实验表明,在四个挑战性专项任务及四个通用基准上均有显著增益。尤其在专项任务中,相比基线,使用LLaVA-1.5-7B和LLaVA-1.5-13B分别取得平均5.4%和4.0%的性能提升。代码已公开于https://github.com/XMUDeepLIT/CVC。

原文摘要 · Abstract (English)

Large Vision-Language Models (LVLMs) have experienced significant advancements in recent years. However, their performance still falls short in tasks requiring deep visual perception, such as identifying subtle differences between images. A potential cause is the scarcity of visual knowledge in popular instruction-tuning corpora, resulting in inadequate visual perception and reasoning capabilities. To address this challenge, we introduce a self-improvement framework grounded in a novel visual knowledge-intensive task, \underline{C}ausality-driven \underline{V}isual object \underline{C}ompletion (CVC). This task requires LVLMs to infer the masked object in an image based on its \textit{causal} relationships with the other visible information. We first obtain rich examples cheaply through our automated instance construction pipeline, without relying on sophisticated LVLMs (\textit{e.g.}, GPT-4V) or human assistance. Then, LVLMs effectively self-improve through trial and error learning using these created instances. Our experiments demonstrate substantial gains across four challenging specialized tasks and four widely-used comprehensive benchmarks. Especially on specialized tasks, our method achieves an average improvement of 5.4\% and 4.0\% compared to the corresponding baselines when utilizing LLaVA-1.5-7B and LLaVA-1.5-13B, respectively. The code is available at https://github.com/XMUDeepLIT/CVC.

视觉语言模型因果推理自训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。