构建临床试验中关键疗效指标的高质量标注数据集。
EvidenceOutcomes: a Dataset of Clinical Trial Publications with Clinically Meaningful Outcomes
- 联合临床医生与NLP专家制定严谨标注规范。
- 500篇文献经三名标注员完成，一致性达0.76。
- 支持机器学习模型训练，提升临床结局提取能力。
循证医学中的证据提取与整合依赖于从生物医学文献中提取PICO（人群、干预、对照、结局）要素。然而，结局作为最复杂的部分，常被现有基准忽略或简化。为此，我们构建了EvidenceOutcomes，一个大规模、高质量的临床有意义结局标注语料库。通过与临床医生和自然语言处理专家反复讨论，制定了稳健的标注指南。三位独立标注员对500篇随机选取的PubMed摘要及140篇来自EBM-NLP语料库的摘要进行结果与结论部分标注，最终获得具有0.76评分一致性的高质量标注数据。此外，基于该数据集微调的PubMedBERT模型在EBM-NLP子集上达到实体级F1=0.69，词元级F1=0.76。EvidenceOutcomes可作为未来机器学习算法开发与评估的共享基准，用于从生物医学摘要中提取临床有意义结局。
原文摘要 · Abstract (English)
The fundamental process of evidence extraction and synthesis in evidence-based medicine involves extracting PICO (Population, Intervention, Comparison, and Outcome) elements from biomedical literature. However, Outcomes, being the most complex elements, are often neglected or oversimplified in existing benchmarks. To address this issue, we present EvidenceOutcomes, a novel, large, annotated corpus of clinically meaningful outcomes extracted from biomedical literature. We first developed a robust annotation guideline for extracting clinically meaningful outcomes from text through iteration and discussion with clinicians and Natural Language Processing experts. Then, three independent annotators annotated the Results and Conclusions sections of a randomly selected sample of 500 PubMed abstracts and 140 PubMed abstracts from the existing EBM-NLP corpus. This resulted in EvidenceOutcomes with high-quality annotations of an inter-rater agreement of 0.76. Additionally, our fine-tuned PubMedBERT model, applied to these 500 PubMed abstracts, achieved an F1-score of 0.69 at the entity level and 0.76 at the token level on the subset of 140 PubMed abstracts from the EBM-NLP corpus. EvidenceOutcomes can serve as a shared benchmark to develop and test future machine learning algorithms to extract clinically meaningful outcomes from biomedical abstracts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。