用大模型自动提取病理报告信息，准确率接近专家水平。
Leveraging large language models for structured information extraction from pathology reports
- 用自然语言指令驱动大模型，零样本完成病理报告结构化提取
- GPT-4o和Llama 3.1 405B准确率达96.1%和94.7%，与人工标注相当
- 开源工具支持非程序员自定义，适合医疗数据标准化研究
从非结构化病理报告中提取结构化信息有助于临床研究的数据可及性。人工提取耗时且成本高，难以扩展。大语言模型（LLMs）通过零样本提示实现高效自动化提取，仅需自然语言指令而无需标注数据或训练。我们评估了大模型在乳腺癌病理报告中提取结构化信息的准确性，与训练过的专业标注员对比。方法：开发了基于LLM的医学报告信息提取工具（Medical Report Information Extractor），构建了金标准数据集，用于评估人类标注员及五种大模型（包括领先的GPT-4o和可本地部署的Llama 3系列）。评估基于111份来自乳腺癌现况研究（BCN Generations Study）的病理报告，提取51项研究数据字典中指定的病理特征。结果：与金标准对比显示，Llama 3.1 405B（94.7%准确率）和GPT-4o（96.1%准确率）的提取准确率与人类标注员（95.4%准确率）无显著差异（p=0.146，p=0.106）；Llama 3.1 70B（91.6%准确率）低于人工水平（p<0.001），但计算开销低，适合本地部署。结论：我们开发了一个开源工具，可通过自然语言定制，模块化设计支持多种提取任务，将非结构化文本转化为标准化结构化数据，提升数据分析的可及性与互操作性。
原文摘要 · Abstract (English)
Background: Structured information extraction from unstructured histopathology reports facilitates data accessibility for clinical research. Manual extraction by experts is time-consuming and expensive, limiting scalability. Large language models (LLMs) offer efficient automated extraction through zero-shot prompting, requiring only natural language instructions without labeled data or training. We evaluate LLMs' accuracy in extracting structured information from breast cancer histopathology reports, compared to manual extraction by a trained human annotator. Methods: We developed the Medical Report Information Extractor, a web application leveraging LLMs for automated extraction. We developed a gold standard extraction dataset to evaluate the human annotator alongside five LLMs including GPT-4o, a leading proprietary model, and the Llama 3 model family, which allows self-hosting for data privacy. Our assessment involved 111 histopathology reports from the Breast Cancer Now (BCN) Generations Study, extracting 51 pathology features specified in the study's data dictionary. Results: Evaluation against the gold standard dataset showed that both Llama 3.1 405B (94.7% accuracy) and GPT-4o (96.1%) achieved extraction accuracy comparable to the human annotator (95.4%; p = 0.146 and p = 0.106, respectively). While Llama 3.1 70B (91.6%) performed below human accuracy (p <0.001), its reduced computational requirements make it a viable option for self-hosting. Conclusion: We developed an open-source tool for structured information extraction that can be customized by non-programmers using natural language. Its modular design enables reuse for various extraction tasks, producing standardized, structured data from unstructured text reports to facilitate analytics through improved accessibility and interoperability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。