评测医疗大模型的患者分诊能力，发现提示词难以解决判断时机错误。
CARE-Bench: Benchmarking Patient-Facing LLM Triage
- 构建500个真实病例的多轮分诊评估基准，每轮输出四类行动标签。
- 未提示模型宏F1仅31.2至50.4，提示后最高达63.4但仍有大量误判。
- 多数模型过早推荐治疗，需追问时仅33.5%正确选择继续询问。
患者面向的医疗大模型日益在医生接触前回答症状问题，其核心安全问题是用户下一步应采取何种行动。本文提出CARE-Bench，一个基于源文本的基准，将患者分诊评估为每轮四类行动的序列任务。该基准包含500个案例和1,059个从医疗对话、咨询及随访中重建的患者描述前缀。我们在269轮保留数据上评估11个模型，在无提示与最小提示的开放式协议下进行测试，使用固定GPT-5.5映射器将每个响应编码为四类行动。未提示模型的宏F1范围为31.2至50.4；提示使10个模型提升，宏F1范围为46.9至63.4，但仍有显著阈值错误。提示模型常在未充分澄清前就推荐诊疗；当正确动作是进一步提问时，仅有33.5%的输出保持了这一步骤。这些错误在提示后仍持续存在，表明患者分诊并非简单提示问题，部署前需明确评估行动时机。
原文摘要 · Abstract (English)
Patient-facing medical LLMs and agents increasingly answer symptom questions before clinician contact, where the key safety question is what action the user should take next. We introduce CARE-Bench, a source-grounded benchmark that evaluates sequential patient-facing triage as a four-label per-turn current-action task. CARE-Bench contains 500 cases and 1,059 evaluated patient-disclosure prefixes reconstructed from medical dialogue, consultation, and follow-up-question sources. We evaluate 11 models on 269 held-out rounds under unprompted and minimally prompted open-ended protocols, using a fixed GPT-5.5 mapper to code each response into the four-label action space. Unprompted macro-F1 remains low, ranging from 31.2 to 50.4. Prompting improves 10 of 11 models, with prompted macro-F1 ranging from 46.9 to 63.4, but substantial threshold errors remain. Prompted models often recommend care before needed clarification is obtained; when the correct action was to ask for more information, only 33.5% of prompted outputs preserved the step. The persistence of these errors after prompting suggests that patient-facing triage is not a simple prompting problem and supports explicit evaluation of action timing before deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。