针对医疗大模型幻觉问题,提出可复用的分析与缓解框架Med-HEAL。
Med-HEAL: Analyzing and Mitigating Hallucinations in Medical LLMs with Hallucination-Aware In-Context Learning

- 基于真实病历数据构建幻觉标注集,结合大模型与医学生双评估
- 自检机制使3/5模型准确率提升,无需更新参数即可生效
- 适合医疗AI安全研究者及临床AI部署团队参考
医疗大语言模型在复杂电子健康记录(EHR)推理中存在幻觉风险,威胁临床决策支持。现有基准缺乏真实临床场景,难以指导实践。本文提出Med-HEAL框架,基于MIMIC-IV出院记录构建的EHRNoteQA基准,通过BioMistral-7B在开放问答任务中生成输出,并采用GPT-4o与医学生联合评估的双重流程进行标注,获得正确性判断与推理错误类型。进一步探索两种缓解策略:自检管道(模型自我审查并重生成)与检索增强的上下文学习(RA-ICL)。在五种开源模型(BioMistral、Llama-3.1、DeepSeek、Qwen2.5、Qwen3)上测试显示,自检策略使其中三款模型准确率显著提升(p < 0.05),且无需参数更新。Med-HEAL提供可复用数据集与实用框架,助力医疗AI安全落地。代码与数据已公开于https://github.com/yimingliao-blad/med-heal.git。
原文摘要 · Abstract (English)
Hallucinations in medical large language models (LLMs) pose serious risks for clinical decision support, particularly when models must reason over complex electronic health records (EHRs). However, existing benchmarks often lack a realistic clinical context and provide limited insight into how hallucinations can be mitigated in practice. We introduce Med-HEAL, a framework for systematically identifying, analyzing, and mitigating hallucinations in medical LLMs using clinically grounded data. Building on the EHRNoteQA benchmark derived from MIMIC-IV discharge summaries, we construct a hallucination dataset by evaluating BioMistral-7B on open-ended clinical question answering tasks. Model outputs are labeled through a dual evaluation pipeline that combines LLM-as-a-Judge assessment (GPT-4o) with human auditing by medical student reviewers, producing correctness judgments and annotations of reasoning errors via a custom web-based evaluation system. We then leverage this dataset to investigate mitigation strategies: a self-critique pipeline, in which the test model reviews its own answers to detect potential errors and regenerates responses for flagged cases, and retrieval-augmented in-context learning (RA-ICL), which exposes the model to hallucinated and corrected examples. Experiments across five open-source LLMs-BioMistral, Llama-3.1, DeepSeek, Qwen2.5, and Qwen3, show that the self-critique strategy improves accuracy for three of five models (p < 0.05) without requiring parameter updates. Med-HEAL provides both a reusable hallucination dataset and a practical framework for studying and mitigating hallucinations in medical LLMs, supporting safer deployment of AI systems in clinical environments. Our code and data are publicly available at https://github.com/yimingliao-blad/med-heal.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。