用多智能体框架精准评估医学报告,兼顾临床细节与可解释性。
GEMA-Score: Granular Explainable Multi-Agent Scoring Framework for Radiology Report Evaluation
- 设计多智能体协作流程,分步解析诊断、位置、严重程度等细粒度信息。
- 在两个公开数据集上与专家评分相关性达0.69和0.45,优于现有方法。
- 提供可读性与术语使用的解释反馈,适合临床可信度评估场景。
自动医学报告生成有望辅助临床诊断、减轻放射科医生负担,并提升诊断一致性。然而,现有评估指标常无法反映生成报告的临床可靠性。早期基于重叠的方法仅关注预测与真实实体的文本匹配,忽略解剖位置、严重程度等细粒度临床信息;部分诊断指标受限于固定词表或模板,难以捕捉多样化的临床表达;基于大模型的方法又缺乏可解释的推理过程,难以在高风险场景中建立信任。为此,本文提出一种细粒度可解释多智能体评分框架(GEMA-Score),通过大模型驱动的多智能体工作流实现客观量化与主观评估。该框架解析结构化报告,利用智能体间交互稳定计算疾病诊断、位置、严重程度与不确定性。同时,一个基于LLM的评分智能体评估报告完整性、可读性及临床术语使用,并提供解释性反馈。大量实验表明,GEMA-Score在公开数据集上与人类专家评分相关性最高,验证了其在临床评分中的有效性(ReXVal数据集Kendall系数=0.69,RadEvalX数据集Kendall系数=0.45)。匿名项目演示见:https://github.com/Zhenxuan-Zhang/GEMA_score。
原文摘要 · Abstract (English)
Automatic medical report generation has the potential to support clinical diagnosis, reduce the workload of radiologists, and demonstrate potential for enhancing diagnostic consistency. However, current evaluation metrics often fail to reflect the clinical reliability of generated reports. Early overlap-based methods focus on textual matches between predicted and ground-truth entities but miss fine-grained clinical details (e.g., anatomical location, severity). Some diagnostic metrics are limited by fixed vocabularies or templates, reducing their ability to capture diverse clinical expressions. LLM-based approaches further lack interpretable reasoning steps, making it hard to assess or trust their behavior in safety-critical settings. These limitations hinder the comprehensive assessment of the reliability of generated reports and pose risks in their selection for clinical use. Therefore, we propose a Granular Explainable Multi-Agent Score (GEMA-Score) in this paper, which conducts both objective quantification and subjective evaluation through a large language model-based multi-agent workflow. Our GEMA-Score parses structured reports and employs stable calculations through interactive exchanges of information among agents to assess disease diagnosis, location, severity, and uncertainty. Additionally, an LLM-based scoring agent evaluates completeness, readability, and clinical terminology while providing explanatory feedback. Extensive experiments validate that GEMA-Score achieves the highest correlation with human expert evaluations on a public dataset, demonstrating its effectiveness in clinical scoring (Kendall coefficient = $0.69$ for ReXVal dataset and Kendall coefficient = $0.45$ for RadEvalX dataset). The anonymous project demo is available at: https://github.com/Zhenxuan-Zhang/GEMA_score.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。