对比传统模型与大模型在影像随访识别中的表现,发现优化后的大模型效果最好。
Identifying Imaging Follow-Up in Radiology Reports: A Comparative Analysis of Traditional ML and LLM Approaches
- 用6393份放射科报告构建标注数据集,评估随访判断能力
- GPT-4o(优化版)F1达0.832,优于其他模型,接近人工水平
- 传统模型如逻辑回归仍有实用价值,适合资源受限场景
大型语言模型(LLMs)在临床自然语言处理中展现出巨大潜力,但针对放射学任务的领域特定数据集仍十分稀缺。本文构建了一个包含6,393份放射科报告的标注语料库,来自586名患者,每份报告均标注了随访影像状态,以支持随访依从性检测系统的开发与基准测试。基于该语料库,我们系统比较了传统机器学习分类器(包括逻辑回归、支持向量机、Longformer)与近期生成式大模型的表现。对于生成式大模型,测试了GPT-4o和开源模型GPT-OSS-20B在两种配置下:基础(Base)与任务优化(Advanced),后者聚焦于元数据、建议句及其上下文。对GPT-OSS-20B进一步优化提示词,显著提升了推理准确性。性能通过精确率、召回率和F1分数评估,并使用非参数自助法估算95%置信区间。各标注者间一致性较高(F1 = 0.846)。GPT-4o(Advanced)表现最佳(F1 = 0.832),紧随其后的是GPT-OSS-20B(Advanced;F1 = 0.828)。逻辑回归与支持向量机也表现良好(F1分别为0.776和0.775),表明尽管通过提示工程可使大模型接近人类水平,但可解释性强且资源高效的模型仍是重要基线。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown considerable promise in clinical natural language processing, yet few domain-specific datasets exist to rigorously evaluate their performance on radiology tasks. In this work, we introduce an annotated corpus of 6,393 radiology reports from 586 patients, each labeled for follow-up imaging status, to support the development and benchmarking of follow-up adherence detection systems. Using this corpus, we systematically compared traditional machine-learning classifiers, including logistic regression (LR), support vector machines (SVM), Longformer, and a fully fine-tuned Llama3-8B-Instruct, with recent generative LLMs. To evaluate generative LLMs, we tested GPT-4o and the open-source GPT-OSS-20B under two configurations: a baseline (Base) and a task-optimized (Advanced) setting that focused inputs on metadata, recommendation sentences, and their surrounding context. A refined prompt for GPT-OSS-20B further improved reasoning accuracy. Performance was assessed using precision, recall, and F1 scores with 95% confidence intervals estimated via non-parametric bootstrapping. Inter-annotator agreement was high (F1 = 0.846). GPT-4o (Advanced) achieved the best performance (F1 = 0.832), followed closely by GPT-OSS-20B (Advanced; F1 = 0.828). LR and SVM also performed strongly (F1 = 0.776 and 0.775), underscoring that while LLMs approach human-level agreement through prompt optimization, interpretable and resource-efficient models remain valuable baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。