用视觉验证提升视觉语言模型的推理能力,避免幻觉和捷径。
Improving Reasoning in Vision-Language Models via Perception Verified Self-Training

- 分离感知与推理,通过图像描述验证确保推理基于真实视觉。
- 在多个数据集上比标准自训练提升最高16%的准确率。
- 适合需要低成本高质量推理的多模态研究者和开发者。
在视觉语言模型中实现类人推理仍是长期挑战。现有方法依赖人工或专有模型生成思维链(CoT),成本高且难扩展。自训练虽具潜力,但常因仅依据答案正确性筛选而产生视觉幻觉和语言捷径。本文提出感知验证自训练框架,强制推理基于视觉事实。方法采用‘描述-推理-结论’的CoT模板,分离感知与推理,实现独立的视觉理解验证。为弥补无真实描述的问题,提出无监督评估方法PerceptEval,基于图像与文本元素对齐度衡量描述质量。结合答案正确性与描述验证,将数据分为易、中、难三类,设计两阶段课程学习:第一阶段在易样本上训练;第二阶段通过已验证描述重构中等难度样本的推理,并仅保留结论正确的样本。该机制确保训练始终基于视觉基础的推理,显著减少幻觉与捷径。跨多个领域和模型的实验表明,相比标准自训练基线,性能最高提升16%,提供了一种无需人工标注思维链的可扩展、低成本解决方案。
原文摘要 · Abstract (English)
Achieving human-like reasoning in Vision-Language Models (VLMs) remains a long-standing challenge. Recent approaches leverage Chain-of-Thought (CoT) rationales generated by human annotators or proprietary models, which are costly and difficult to scale. Self-training offers a promising alternative but often suffers from visual hallucinations and language shortcuts because rationales are filtered only by answer correctness without verifying visual perception. We propose a perception-verified self-training framework that enforces visually grounded reasoning. Our method employs a CoT template (caption-reasoning-conclusion) that disentangles perception from reasoning, enabling independent verification of visual understanding. To compensate for the absence of ground-truth captions, we introduce PerceptEval, an unsupervised method that evaluates caption quality based on its alignment with visual and textual elements in the image. Using caption verification together with answer correctness, we partition the data into easy, medium, and hard subsets and design a two-stage curriculum learning strategy. Stage 1 trains on easy samples, while Stage 2 enhances medium samples by regenerating reasoning conditioned on verified captions and retaining only those with correct conclusions. This ensures training is performed exclusively on perceptually grounded reasoning, reducing hallucinations and language shortcuts. Extensive experiments across diverse domains and models demonstrate improvements of up to 16% over standard self-training baselines, showing that our framework provides a scalable and cost-effective solution for advancing multimodal reasoning without manually annotated CoT rationales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。