不训练模型,用解剖证据抑制医学多模态大模型幻觉
Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence

- 通过解剖区域先验引导视觉感知与文本推理双路径校准
- 在5个数据集上提升闭合问答准确率6%,开放问答幻觉降低35%
- 适合需要可验证医疗决策的AI系统开发者使用
多模态大语言模型在临床视觉问答和放射科报告生成中表现优异,但推理时仍存在与影像证据不符的幻觉问题。现有缓解方法多依赖额外训练、外部检索或分步后处理,增加成本且泛化性差。为此,我们提出无需训练的协同感知-推理治理框架,通过双路径注入可验证解剖证据:利用MedSAM获取感兴趣区域先验,以区域引导激活调节视觉感知轨迹;将解剖坐标映射为离散语义标记,作为可验证的外部记忆锚定文本推理轨迹。引入任务感知动态路由,根据任务语义选择模态特定干预策略,在感知真实性与语言流畅性间取得平衡。在2项任务、5个数据集上评估了LLaVA-1.5-7B、LLaVA-Med-1.5-7B、Qwen3-VL-8B/32B、InternVL-3.5-8B/38B模型。受控消融与可视化验证框架有效性,其在医疗基准上持续优于基线,闭合问答准确率最高提升约6%,开放问答幻觉减少约35%。代码已开源。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) show strong promise for clinical VQA and radiology report generation, yet inference-time hallucinations still undermine trustworthy use: models can produce fluent conclusions that conflict with imaging evidence. Existing mitigation strategies typically rely on additional training, external retrieval/knowledge bases, or multi-stage post-hoc verification, which increases cost and pipeline complexity and often generalizes poorly across models and tasks.To address this, we propose a holistic, training-free evidence-injection framework that systematically mitigates hallucinations through dual-side evidence injection. By leveraging ROI priors acquired using MedSAM in our implementation, we recalibrate the visual perception trajectory via ROI-guided activation modulation while anchoring the textual reasoning trajectory by mapping anatomical coordinates into discrete semantic tokens as verifiable external memory. Then we introduce a task-aware dynamic router to select modality-specific interventions based on task semantics, balancing perceptual grounding and linguistic fluency. We conduct systematic evaluations on 2 tasks and 5 datasets using \texttt{LLaVA-1.5-7B}, \texttt{LLaVA-Med-1.5-7B}, \texttt{Qwen3-VL-8B/32B}, and \texttt{InternVL-3.5-8B/38B}. Controlled ablations and visualizations further validate the framework, which consistently outperforms baselines across medical benchmarks, improving close-ended accuracy by up to $\sim\mathbf{6}\%\uparrow$ and reducing open-ended hallucinations by $\sim\mathbf{35}\%\downarrow$. The code has been made available on GitHub: \href{https://github.com/Henry991115/SPRG}{\textcolor{blue}{https://github.com/Henry991115/SPRG}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。