用配对排序构建医疗消息紧急程度评估基准，提升患者门户消息优先级判断。
Medical Triage as Pairwise Ranking: A Benchmark for Urgency in Patient Portal Messages
- 将医疗分诊转化为消息间紧急程度对比任务，模拟医生邮箱排序场景。
- 在1569条真实消息上构建2000+高质量测试对，8B模型性能提升15-16点。
- 适合医疗AI、临床决策支持系统研究者，推动智能分诊发展。
医疗分诊是根据医疗需求分配资源并优先处理患者的任务。本文首次提出大规模公开数据集，用于研究异步门诊门户消息中的医疗分诊问题。我们创新性地将患者消息分诊定义为配对推理任务，训练大语言模型在头对头比较中判断哪条消息更紧急，模拟医生邮箱的重新排序过程。新基准PMR-Bench包含1569条唯一消息和2000+高质量测试对，支持配对式医疗紧急程度评估，并提供可扩展的训练数据生成流程。数据集融合了非结构化患者撰写消息与真实电子健康记录（EHR）数据，贴近真实临床场景。我们设计了一种自动化数据标注策略，为大模型提供领域内指导。基于此数据，训练出UrgentReward和UrgentSFT两类模型，分别采用Bradley-Terry和下一标记预测目标进行配对紧急程度分类。结果表明，UrgentSFT在PMR-Bench上表现最佳，而UrgentReward在低资源条件下更具优势。例如，UrgentSFT-8B和UrgentReward-8B相比现成的8B模型，在邮箱排序指标上分别提升15和16个百分点。论文资源详见：https://tinyurl.com/Patient-Message-Triage
原文摘要 · Abstract (English)
Medical triage is the task of allocating medical resources and prioritizing patients based on medical need. This paper introduces the first large-scale public dataset for studying medical triage in the context of asynchronous outpatient portal messages. Our novel task formulation views patient message triage as a pairwise inference problem, where we train LLMs to choose `"which message is more medically urgent" in a head-to-head tournament-style re-sort of a physician's inbox. Our novel benchmark PMR-Bench contains 1569 unique messages and 2,000+ high-quality test pairs for pairwise medical urgency assessment alongside a scalable training data generation pipeline. PMR-Bench includes samples that contain both unstructured patient-written messages alongside real electronic health record (EHR) data, emulating a real-world medical triage scenario. We develop a novel automated data annotation strategy to provide LLMs with in-domain guidance on this task. The resulting data is used to train two model classes, UrgentReward and UrgentSFT, leveraging Bradley-Terry and next token prediction objective, respectively to perform pairwise urgency classification. We find that UrgentSFT achieves top performance on PMR-Bench, with UrgentReward showing distinct advantages in low-resource settings. For example, UrgentSFT-8B and UrgentReward-8B provide a 15- and 16-point boost, respectively, on inbox sorting metrics over off-the-shelf 8B models. Paper resources can be found at https://tinyurl.com/Patient-Message-Triage
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。