测试大模型能否用血压数据回答医学问题，发现大模型表现更好。
BPQA Dataset: Evaluating How Well Language Models Leverage Blood Pressures to Answer Biomedical Questions
- 构建血压问答数据集BPQA，验证模型对临床测量值的利用能力。
- GPT-3.5和MedAlpaca因模型规模更大，更擅长使用血压数据。
- 为小模型添加标签信息可提升性能，适合医疗领域模型优化研究者。
临床指标如血压和呼吸频率在诊断与患者监护中至关重要，是生物医学数据的重要组成部分，可用于训练基于Transformer的语言模型（LMs）以改善医疗交付。然而，尚不清楚语言模型是否能有效解读并利用这些临床测量数据。本文聚焦血压读数这一常规监测指标，探究两个问题：第一，语言模型能否有效利用血压数据回答相关医学问题？第二，如何提升涉及测量值的医学问答任务性能？我们构建了新数据集BPQA（Blood Pressure Question Answering），包含100个经医学生验证的医学问答对，其答案依赖于血压数据。评估了四种语言模型：BERT、BioBERT、MedAlpaca和GPT-3.5。结果表明，较大模型（GPT-3.5和MedAlpaca）相比小型模型（BERT和BioBERT）更能从血压数据中获益；此外，为生物医学专用模型（BioBERT和MedAlpaca）引入测量标签可进一步提升性能，提示检索机制或有助于增强领域特定模型的表现。
原文摘要 · Abstract (English)
Clinical measurements such as blood pressures and respiration rates are critical in diagnosing and monitoring patient outcomes. It is an important component of biomedical data, which can be used to train transformer-based language models (LMs) for improving healthcare delivery. It is, however, unclear whether LMs can effectively interpret and use clinical measurements. We investigate two questions: First, can LMs effectively leverage clinical measurements to answer related medical questions? Second, how to enhance an LM's performance on medical question-answering (QA) tasks that involve measurements? We performed a case study on blood pressure readings (BPs), a vital sign routinely monitored by medical professionals. We evaluated the performance of four LMs: BERT, BioBERT, MedAlpaca, and GPT-3.5, on our newly developed dataset, BPQA (Blood Pressure Question Answering). BPQA contains $100$ medical QA pairs that were verified by medical students and designed to rely on BPs . We found that GPT-3.5 and MedAlpaca (larger and medium sized LMs) benefit more from the inclusion of BPs than BERT and BioBERT (small sized LMs). Further, augmenting measurements with labels improves the performance of BioBERT and Medalpaca (domain specific LMs), suggesting that retrieval may be useful for improving domain-specific LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。