用大模型替代生物医学文本标注员,提升效率并减少人工依赖。
Can Frontier LLMs Replace Annotators in Biomedical Text Mining? Analyzing Challenges and Exploring Solutions
- 针对标注数据隐含特征、格式限制和指南遵循难题,设计动态提示与指令提取流程。
- 前沿大模型在无需微调下性能接近或超过现有最佳BERT模型。
- 通过合成数据训练的轻量级模型也能达到实用水平,适合生产落地。
先前研究显示大模型在生物医学文本挖掘中表现不佳。通过分析其失败模式,我们识别出三大挑战:(1)大模型难以从监督数据中学习特定数据集的隐含特征;(2)判别任务常见的格式要求限制了大模型的推理能力,尤其对无测试时计算资源的模型;(3)大模型难以遵循标注指南与精确模式,影响其理解复杂标注需求。我们针对上述问题设计了提示工程方法,并开发了一种从标注指南中动态提取指令的流水线。结果表明,前沿大模型可在无需微调且极少人工标注数据的情况下,达到或超越当前最优的BERT模型性能。此外,我们对闭源大模型进行了模型蒸馏,证明仅使用大模型生成的合成数据训练的BERT模型也能实现可应用的性能。基于这些发现,我们探讨了在生物医学文本挖掘生产场景中部分替代人工标注的可行性。
原文摘要 · Abstract (English)
Multiple previous studies have reported suboptimal performance of LLMs in biomedical text mining. By analyzing failure patterns in these evaluations, we identified three primary challenges for LLMs in biomedical corpora: (1) LLMs fail to learn implicit dataset-specific nuances from supervised data, (2) The common formatting requirements of discriminative tasks limit the reasoning capabilities of LLMs particularly for LLMs that lack test-time compute, and (3) LLMs struggle to adhere to annotation guidelines and match exact schemas, which hinders their ability to understand detailed annotation requirements which is essential in biomedical annotation workflow. We experimented with prompt engineering techniques targeted to the above issues, and developed a pipeline that dynamically extracts instructions from annotation guidelines. Our results show that frontier LLMs can approach or surpass the performance of SOTA BERT-based models with minimal reliance on manually annotated data and without fine-tuning. Furthermore, we performed model distillation on a closed-source LLM, demonstrating that a BERT model trained exclusively on synthetic data annotated by LLMs can also achieve a practical performance. Based on these findings, we explored the feasibility of partially replacing manual annotation with LLMs in production scenarios for biomedical text mining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。