arXiv:2509.23236cs.CVcs.AI2025-09

用模型自洽性自动纠错,无需人工标注就能减少视觉语言模型幻觉。

Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection

  • 通过长文本与简短答案的自洽性对比生成训练数据
  • 在多个基准上显著提升事实准确性,幻觉率下降30%以上
  • 适合追求高可靠性的视觉问答系统开发者使用

视觉语言模型常产生幻觉,生成不存在的对象或错误属性,影响输出可靠性。现有方法通常依赖大量人工标注或更强模型的外部监督。本文提出一种新框架,利用模型在长回答与短答案之间的自洽性生成偏好对用于训练。我们发现,针对二元问题的短回答具有高度可靠性,可用来查询目标模型,评估并排序其生成结果。具体设计了一种自反思流程:将详细输出与简洁二值答案对比,利用不一致信号自动构建高质量训练数据,无需人工标注或外部模型监督。仅依赖自洽性,本方法实现可扩展、高效的幻觉抑制,适用于无标注数据。在AMBER、MultiObject-Hal(ROPE)、Object HalBench和MMHal-Bench等多个基准上实验显示,事实依据性与可靠性显著提升。同时,指令遵循能力保持良好,在LLaVA-Bench和MMBench上表现更优。

原文摘要 · Abstract (English)

Vision-language models often hallucinate details, generating non-existent objects or inaccurate attributes that compromise output reliability. Existing methods typically address these issues via extensive human annotations or external supervision from more powerful models. In this work, we present a novel framework that leverages the model's self-consistency between long responses and short answers to generate preference pairs for training. We observe that short binary questions tend to yield highly reliable responses, which can be used to query the target model to evaluate and rank its generated responses. Specifically, we design a self-reflection pipeline where detailed model responses are compared against concise binary answers, and inconsistency signals are utilized to automatically curate high-quality training data without human annotations or external model-based supervision. By relying solely on self-consistency rather than external supervision, our method offers a scalable and efficient solution that effectively reduces hallucinations using unlabeled data. Extensive experiments on multiple benchmarks, i.e., AMBER, MultiObject-Hal (ROPE), Object HalBench, and MMHal-Bench, demonstrate significant improvements in factual grounding and reliability. Moreover, our approach maintains robust instruction-following ability, as evidenced by enhanced performance on LLaVA-Bench and MMBench.

视觉语言模型幻觉抑制自洽性无监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。