arXiv:2603.02556cs.CVcs.AI2026-03中稿 · ICLR被引 8

利用视觉对比提升VLM推理能力,减少幻觉。

Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs

  • 通过对比图像对引导模型生成更准确的推理路径。
  • 构建了5.5万条样本的VisCoR-55K数据集,显著提升推理性能。
  • 适合研究视觉语言模型、推理增强与自改进方法的学者。

推理已成为大语言模型的关键能力。在语言任务中,可通过自我改进技术优化推理路径以用于后续微调。然而,将此类基于语言的自改进方法扩展到视觉语言模型(VLMs)面临独特挑战:模型生成的推理路径中的视觉幻觉难以验证或修正。本文观察到一个重要现象:当面对一对视觉相似但问题同义的对比性视觉问答(VQA)样本时,VLM能更精准地识别相关视觉线索。基于此,提出视觉对比自教学推理框架(VC-STaR),利用视觉对比缓解模型生成推理路径中的幻觉。我们收集多样化的VQA数据集,根据多模态相似性筛选对比对,并使用VC-STaR生成推理过程,最终构建新视觉推理数据集VisCoR-55K。该数据集通过监督微调显著提升了多种VLM的推理能力。大量实验表明,VC-STaR不仅优于现有自改进方法,还超越在当前最先进视觉推理数据集上微调的模型,证明VLM固有的对比能力可自我驱动其视觉推理能力提升。项目地址:https://github.com/zhiyupan42/VC-STaR。

原文摘要 · Abstract (English)

Reasoning has emerged as a key capability of large language models. In linguistic tasks, this capability can be enhanced by self-improving techniques that refine reasoning paths for subsequent finetuning. However, extending these language-based self-improving approaches to vision language models (VLMs) presents a unique challenge:~visual hallucinations in reasoning paths cannot be effectively verified or rectified. Our solution starts with a key observation about visual contrast: when presented with a contrastive VQA pair, i.e., two visually similar images with synonymous questions, VLMs identify relevant visual cues more precisely. Motivated by this observation, we propose Visual Contrastive Self-Taught Reasoner (VC-STaR), a novel self-improving framework that leverages visual contrast to mitigate hallucinations in model-generated rationales. We collect a diverse suite of VQA datasets, curate contrastive pairs according to multi-modal similarity, and generate rationales using VC-STaR. Consequently, we obtain a new visual reasoning dataset, VisCoR-55K, which is then used to boost the reasoning capability of various VLMs through supervised finetuning. Extensive experiments show that VC-STaR not only outperforms existing self-improving approaches but also surpasses models finetuned on the SoTA visual reasoning datasets, demonstrating that the inherent contrastive ability of VLMs can bootstrap their own visual reasoning. Project at: https://github.com/zhiyupan42/VC-STaR.

视觉推理自改进对比学习VLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。