用LLaMA 3做生物医学多跳问答,提升理解但难拿满分
CaresAI at BioCreative IX Track 1 -- LLM for Biomedical QA
- 用生物医学数据微调LLaMA 3,支持多跳推理
- 概念准确率最高达0.8,但精确匹配分仍偏低
- 设计两阶段推理流程,改善答案格式与精度
大型语言模型在多个领域已展现出精准问答能力。然而,在部署于真实生物医学和医疗应用前,对其复杂问答能力进行严格评估至关重要。本文介绍我们在BioCreative IX共享任务中MedHopQA赛道的方法,该任务聚焦涉及疾病、基因和化学物质的多跳生物医学问答。我们采用监督微调策略,基于LLaMA 3 8B模型,并使用来自BioASQ、MedQuAD和TREC等外部来源的生物医学问答数据集进行训练。探索了三种实验设置:联合微调短答案与长答案、仅微调短答案、仅微调长答案。尽管模型展现出较强的领域理解能力,概念级准确率最高达0.8,但在测试阶段其精确匹配(EM)得分仍显著偏低。为此,我们引入两阶段推理流程以精确提取短答案,缓解冗余并提升与评估指标的一致性。尽管部分改进,生成严格格式化输出的挑战依然存在。研究结果凸显了生物医学大模型在语义理解与精确答案评估之间的差距,推动进一步研究输出控制与后处理策略。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly evident for accurate question answering across various domains. However, rigorous evaluation of their performance on complex question-answering (QA) capabilities is essential before deployment in real-world biomedical and healthcare applications. This paper presents our approach to the MedHopQA track of the BioCreative IX shared task, which focuses on multi-hop biomedical question answering involving diseases, genes, and chemicals. We adopt a supervised fine-tuning strategy leveraging LLaMA 3 8B, enhanced with a curated biomedical question-answer dataset compiled from external sources including BioASQ, MedQuAD, and TREC. Three experimental setups are explored: fine-tuning on combined short and long answers, short answers only, and long answers only. While our models demonstrate strong domain understanding, achieving concept-level accuracy scores of up to 0.8, their Exact Match (EM) scores remain significantly lower, particularly in the test phase. We introduce a two-stage inference pipeline for precise short-answer extraction to mitigate verbosity and improve alignment with evaluation metrics. Despite partial improvements, challenges persist in generating strictly formatted outputs. Our findings highlight the gap between semantic understanding and exact answer evaluation in biomedical LLM applications, motivating further research in output control and post-processing strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。