用大模型和实体识别，自动评估放射科报告的语义相似度。
Semantic Similarity in Radiology Reports via LLMs and NER
- 结合Llama 3.1与命名实体识别，加权计算报告相似度。
- 相比人工评分，相似度判断准确率达93%（±1误差内）。
- 可解释得分帮助新手医生发现报告中的知识盲区。
放射科报告评估是放射科医生培训的关键环节，对确保诊断准确性至关重要。通常由初级医生撰写初稿，再由资深医生修改定稿。识别初稿与终稿间的语义差异，对新人训练和发现临床知识缺口意义重大。尽管医学AI发展迅速，但大语言模型（LLMs）在该领域应用受限于专业领域知识需求。本文探索了LLMs在放射科报告对比中的表现，并评估了传统命名实体识别（NER）方法。两者均存在反馈不精准的局限。为此，我们提出Llama-EntScore：融合Llama 3.1与NER，通过可调权重突出或弱化特定类型差异，生成量化相似度分数并提供可解释性反馈。实验表明，该方法在与放射科医生标注的真值对比中，精确匹配率达67%，±1误差内准确率达93%，优于独立使用LLM或NER。代码已开源。
原文摘要 · Abstract (English)
Radiology report evaluation is a crucial part of radiologists' training and plays a key role in ensuring diagnostic accuracy. As part of the standard reporting workflow, a junior radiologist typically prepares a preliminary report, which is then reviewed and edited by a senior radiologist to produce the final report. Identifying semantic differences between preliminary and final reports is essential for junior doctors, both as a training tool and to help uncover gaps in clinical knowledge. While AI in radiology is a rapidly growing field, the application of large language models (LLMs) remains challenging due to the need for specialised domain knowledge. In this paper, we explore the ability of LLMs to provide explainable and accurate comparisons of reports in the radiology domain. We begin by comparing the performance of several LLMs in comparing radiology reports. We then assess a more traditional approach based on Named-Entity-Recognition (NER). However, both approaches exhibit limitations in delivering accurate feedback on semantic similarity. To address this, we propose Llama-EntScore, a semantic similarity scoring method using a combination of Llama 3.1 and NER with tunable weights to emphasise or de-emphasise specific types of differences. Our approach generates a quantitative similarity score for tracking progress and also gives an interpretation of the score that aims to offer valuable guidance in reviewing and refining their reporting. We find our method achieves 67% exact-match accuracy and 93% accuracy within +/- 1 when compared to radiologist-provided ground truth scores - outperforming both LLMs and NER used independently. Code is available at: https://github.com/otmive/llama_reports
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。