arXiv:2603.29373cs.CL2026-03被引 3

测试大模型在真实复杂医患对话中的表现,发现常见问题导致安全风险。

Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultations

  • 构建四类真实医患互动挑战行为的评测基准
  • 模型在矛盾或不实信息下错误率超60%且无法识别自诊
  • 适合医疗AI安全研究者与临床对话系统开发者参考

大型语言模型在医疗咨询和健康信息支持中应用日益广泛。然而,现有评估大多假设患者提问清晰规范,缺乏现实性。本文研究真实诊疗中常见的四种挑战性患者行为:信息矛盾、事实错误、自我诊断和抗拒治疗,并定义了对应的不安全响应判别标准。基于四个现有医学对话数据集,我们构建了涵盖692个双语(英/中)多轮对话的CPB-Bench评测基准。评估结果显示,尽管整体表现尚可,但模型在处理矛盾或医学上不合理的患者输入时存在显著缺陷,尤其难以识别自诊行为。我们进一步测试四种干预策略,发现其改善效果不一致,且可能引入不必要的纠正。相关数据与代码已开源。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly used for medical consultation and health information support. In this high-stakes setting, safety depends not only on medical knowledge, but also on how models respond when patient inputs are unclear, inconsistent, or misleading. However, most existing medical LLM evaluations assume idealized and well-posed patient questions, which limits their realism. In this paper, we study challenging patient behaviors that commonly arise in real medical consultations and complicate safe clinical reasoning. We define four clinically grounded categories of such behaviors: information contradiction, factual inaccuracy, self-diagnosis, and care resistance. For each behavior, we specify concrete failure criteria that capture unsafe responses. Building on four existing medical dialogue datasets, we introduce CPB-Bench (Challenging Patient Behaviors Benchmark), a bilingual (English and Chinese) benchmark of 692 multi-turn dialogues annotated with these behaviors. We evaluate a range of open- and closed-source LLMs on their responses to challenging patient utterances. While models perform well overall, we identify consistent, behavior-specific failure patterns, with particular difficulty in handling contradictory or medically implausible patient information. We also study four intervention strategies and find that they yield inconsistent improvements and can introduce unnecessary corrections. We release the dataset and code.

医疗AILLM评测对话安全真实场景

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。