Kestrel通过视觉证据验证实现无训练幻觉抑制,提升大模型推理可靠性。
Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigation
- 引入视觉定位代理与证据验证机制,生成可复用的结构化文本证据。
- 在POPE上平均提升3.31%,MME-Hallucination上提升28.34%(Qwen3-VL)。
- 提供透明验证路径,适合需可解释性的多模态应用开发与研究者。
大型视觉语言模型(LVLMs)在多模态任务中表现日益强大,但依然存在严重幻觉问题,限制其实际部署。由于大规模模型训练以避免幻觉成本过高,无训练方法成为低成本且灵活的替代方案,但现有基于解码或工具调用的方法效果有限且可解释性弱。本文提出Kestrel,一种无训练的LVLM幻觉缓解框架,结合显式视觉定位代理与证据验证自修正机制。具体而言,Kestrel首先收集显式视觉证据,并将工具输出转化为可复用、结构化的文本证据;其次,利用LVLM判别器对证据进行验证,再基于验证后的证据迭代自修正答案,降低过度修正风险。大量实验表明,Kestrel在多个幻觉评估基准上超越强基线模型:例如在POPE上平均提升+3.31%,在MME-Hallucination上提升+28.34%(使用Qwen3-VL)。同时,该方法提供透明的验证轨迹,支持幻觉诊断与分析——集成的自修正模块与定位代理分别在POPE上带来平均+2.0%的增益。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) have become increasingly strong but remain prone to hallucinations in multimodal tasks, which significantly narrows their deployment. As training these LVLMs to avoid hallucinations becomes prohibitively expensive for larger models, training-free methods offer a cheap and flexible solution to this problem, yet existing approaches based on decoding or tool use often bring limited gains and/or weak interpretability. We propose Kestrel, a training-free framework for LVLM hallucination mitigation that combines an explicit visual-grounding agent with evidence-verified self-refinement mechanism. In detail, Kestrel first collects explicit visual evidence and converts tool outputs into reusable and structured textual evidence. Second, to take full advantage of these evidence, Kestrel verifies them via an LVLM judge for evidence checking, then iteratively self-refine answers based on verified evidence to reduce the risk of over-correction. Extensive experiments show that Kestrel improves performance over strong baselines across hallucination benchmarks (e.g., average +3.31% on POPE and +28.34 on MME-Hallucination with Qwen3-VL), while providing transparent verification traces for hallucination diagnosis and analysis -- e.g., both the integrated self-refinement module and grounding agent contributing an average +2.0% gain on POPE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。