用混合模型自动分类癌症报告,准确率超90%,省下近千人工时。
ELM: A Hybrid Ensemble of Language Models for Automated Tumor Group Classification in Population-Based Cancer Registries
- 组合六个小模型+大模型,分段分析报告并投票决策。
- 测试集上精确率与召回率均达0.94,显著优于传统方法。
- 适合医疗数据自动化处理场景,尤其擅长复杂肿瘤类型分类。
基于人群的癌症登记系统(PBCRs)需从非结构化病理报告中手动提取数据,每年约10万份报告需900人时完成肿瘤分组。现有规则系统难以应对语言复杂性。本文提出ELM(语言模型集成),结合微调的小型编码器模型与大语言模型(LLM)。ELM采用六模型集成:三个分析报告上部,三个分析下部,以覆盖更多文本(受令牌限制)。当至少五模型一致时确定分组;否则由LLM通过精心设计提示在可能类别中仲裁。在2058份涵盖19类肿瘤的测试集上,ELM加权精确率与召回率均为0.94,显著优于仅编码器模型(F1: 0.91,p<0.001),大幅超越规则系统。对白血病(F1: 0.76→0.88)、淋巴瘤(0.76→0.89)、皮肤癌(0.44→0.58)等难点类别提升明显。已在不列颠哥伦比亚癌症登记处上线,使人工审核减少60-70%,年节省约900人时,保持数据质量。这是首个在真实世界登记系统中成功部署的混合小模型-大模型架构,证明合理组合可兼顾高精度与效率。
原文摘要 · Abstract (English)
Background: Population-based cancer registries (PBCRs) manually extract data from unstructured pathology reports, a labor-intensive process where assigning reports to tumor groups can consume 900 person-hours annually for approximately 100,000 reports at a medium-sized registry. Current automated rule-based systems fail to handle the linguistic complexity of this classification task. Materials and Methods: We present ELM (Ensemble of Language Models), a novel hybrid approach combining small, encoder only language models and large language models (LLMs). ELM employs an ensemble of six fine-tuned encoder only models: three analyzing the top portion and three analyzing the bottom portion of each report to maximize text coverage given token limits. A tumor group is assigned when at least five of six models agree; otherwise, an LLM arbitrates using a carefully curated prompt constrained to likely tumor groups. Results: On a held-out test set of 2,058 pathology reports spanning 19 tumor groups, ELM achieves weighted precision and recall of 0.94, representing a statistically significant improvement (p<0.001) over encoder-only ensembles (0.91 F1-score) and substantially outperforming rule-based approaches. ELM demonstrates particular gains for challenging categories including leukemia (F1: 0.76 to 0.88), lymphoma (0.76 to 0.89), and skin cancer (0.44 to 0.58). Discussion: Deployed in production at British Columbia Cancer Registry, ELM has reduced manual review requirements by approximately 60-70%, saving an estimated 900 person-hours annually while maintaining data quality standards. Conclusion: ELM represents the first successful deployment of a hybrid small, encoder only models-LLM architecture for tumor group classification in a real-world PBCR setting, demonstrating how strategic combination of language models can achieve both high accuracy and operational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。