提出新型评估框架VERT,提升医学影像报告的AI评判可靠性。
VERT: Reliable LLM Judges for Radiology Report Evaluation
- 设计VERT框架,系统对比多种LLM评测方法与配置。
- 相比GREEN指标,VERT在专家评分相关性上提升11.7%。
- 仅用1300样本微调即达25%性能增益,推理速度加快37.2倍。
现有医学影像报告评估研究多聚焦于胸部X光的LLM指标设计与小模型微调,但其在其他模态和解剖部位上的鲁棒性尚不明确。本研究通过全面相关性分析,比较了现有三种LLM作为裁判的指标(RadFact、GREEN、FineRadScore)及提出的VERT框架,涵盖开源与闭源模型(含推理型与非推理型)、不同规模模型,在两个专家标注数据集RadEval与RaTE-Eval上进行评估,覆盖多种模态与解剖部位。进一步在RaTE-Eval上测试少样本学习、集成策略与参数高效微调。通过系统性错误检测与分类,分析各指标与专家判断的一致性,识别高/低一致区域。结果表明,VERT相较GREEN最高提升11.7%相关性;对Qwen3-30B进行微调仅需1,300样本,性能最高提升25%,且推理时间最多降低37.2倍。研究证实了基于LLM的评估有效性,展示了轻量适配即可实现可靠评价。
原文摘要 · Abstract (English)
Current literature on radiology report evaluation has focused primarily on designing LLM-based metrics and fine-tuning small models for chest X-rays. However, it remains unclear whether these approaches are robust when applied to reports from other modalities and anatomies. Which model and prompt configurations are best suited to serve as LLM judges for radiology evaluation? We conduct a thorough correlation analysis between expert and LLM-based ratings. We compare three existing LLM-as-a-judge metrics (RadFact, GREEN, and FineRadScore) alongside VERT, our proposed LLM-based metric, using open- and closed-source models (reasoning and non-reasoning) of different sizes across two expert-annotated datasets, RadEval and RaTE-Eval, spanning multiple modalities and anatomies. We further evaluate few-shot approaches, ensembling, and parameter-efficient fine-tuning using RaTE-Eval. To better understand metric behavior, we perform a systematic error detection and categorization study to assess alignment of these metrics against expert judgments and identify areas of lower and higher agreement. Our results show that VERT improves correlation with radiologist judgments by up to 11.7% relative to GREEN. Furthermore, fine-tuning Qwen3 30B yield gains of up to 25% using only 1,300 training samples. The fine-tuned model also reduces inference time up to 37.2 times. These findings highlight the effectiveness of LLM-based judges and demonstrate that reliable evaluation can be achieved with lightweight adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。