用视觉依赖分数抑制医学大模型幻觉，让回答更可信。
VGS-Decoding: Visual Grounding Score Guided Decoding for Hallucination Mitigation in Medical VLMs
- 通过对比原始与失真图像的生成分布，计算每个词的视觉依赖度。
- 在解码时动态增强有视觉依据的词，抑制无根据的生成，提升准确率。
- 无需训练、开销小，适合临床场景部署，对多种医学模型都有效。
医学视觉语言模型常因依赖语言先验而非视觉证据而产生幻觉，危及临床应用。本文提出无需训练的推理阶段幻觉抑制方法VGS-Decoding：核心思想是幻觉词汇在图像退化后概率不变或上升，而视觉相关词汇概率下降。通过引入视觉接地分数（VGS），量化每个词对视觉信息的依赖程度，在解码时动态重加权，强化视觉支撑项，抑制幻觉项。相比固定权重对比方法，本方法实现逐词自适应控制。在MIMIC-Diff-VQA和VQA-RAD数据集上，对LLaVA-Med、CheXagent和MedGemma均取得稳定提升，最高整体性能提升9.12%，开放问答召回率提升8.98%，推理开销仅增加2倍，且无需额外训练，具备临床部署可行性。论文被接收后将公开代码以促进可复现性。
原文摘要 · Abstract (English)
Medical Vision-Language Models (VLMs) often hallucinate by generating responses based on language priors rather than visual evidence, posing risks in clinical applications. We propose Visual Grounding Score Guided Decoding (VGS-Decoding), a training-free method to mitigate hallucinations during inference. Our key insight is that hallucinated tokens maintain or increase their probability when visual information is degraded, while visually grounded tokens decrease in probability. We introduce the Visual Grounding Score (VGS), which measures each token's visual dependency by comparing distributions from original and distorted images. During decoding, we reweight probabilities by amplifying visually grounded tokens while suppressing hallucinations. Unlike fixed-weight contrastive methods, VGS-Decoding provides per-token adaptive control. Experiments on MIMIC-Diff-VQA and VQA-RAD across LLaVA-Med, CheXagent, and MedGemma demonstrate consistent improvements, with up to +9.12% overall gain and $+8.98\%$ in open-ended recall, while introducing only $2\times$ inference overhead and no additional training, making it practical for clinical deployment. Upon acceptance, code will be released publicly to facilitate reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。