通过语义一致性量化不确定性,提升放射科报告生成的准确性。
Semantic Consistency-Based Uncertainty Quantification for Factuality in Radiology Report Generation
- 基于语义一致性构建无需修改模型的不确定性评估框架。
- 拒绝20%高不确定报告后,事实性得分提升10%。
- 可精准定位低准确率句子,成功率超82.9%。
放射科报告生成(RRG)在自动化报告撰写方面展现出巨大潜力,但确保生成内容的事实正确性仍是关键挑战。尽管已有视觉-语言大模型(VLLM)尝试解决此问题,仍易产生幻觉并输出错误诊断信息。为此,本文提出一种基于语义一致性的不确定性量化框架,支持报告级与句子级不确定性评估。该方法无需修改底层模型或访问其内部状态(如输出词元概率),可作为即插即用模块集成于现有先进模型。大量实验表明,该方法能有效检测幻觉并提升生成报告的事实性。通过拒绝20%高不确定性报告,使用Radialog模型在MIMIC-CXR数据集上使事实性得分提升10%;句级不确定性可将每份报告中精度最低的句子标记出,成功率达82.9%。代码已开源:https://github.com/BU-DEPEND-Lab/SCUQ-RRG。
原文摘要 · Abstract (English)
Radiology report generation (RRG) has shown great potential in assisting radiologists by automating the labor-intensive task of report writing. While recent advancements have improved the quality and coherence of generated reports, ensuring their factual correctness remains a critical challenge. Although generative medical Vision Large Language Models (VLLMs) have been proposed to address this issue, these models are prone to hallucinations and can produce inaccurate diagnostic information. To address these concerns, we introduce a novel Semantic Consistency-Based Uncertainty Quantification framework that provides both report-level and sentence-level uncertainties. Unlike existing approaches, our method does not require modifications to the underlying model or access to its inner state, such as output token logits, thus serving as a plug-and-play module that can be seamlessly integrated with state-of-the-art models. Extensive experiments demonstrate the efficacy of our method in detecting hallucinations and enhancing the factual accuracy of automatically generated radiology reports. By abstaining from high-uncertainty reports, our approach improves factuality scores by $10$\%, achieved by rejecting $20$\% of reports using the \texttt{Radialog} model on the MIMIC-CXR dataset. Furthermore, sentence-level uncertainty flags the lowest-precision sentence in each report with an $82.9$\% success rate. Our implementation is open-source and available at https://github.com/BU-DEPEND-Lab/SCUQ-RRG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。