arXiv:2603.11281cs.CL2026-03被引 2

真实患者问诊对话构建多轮医学问答基准,揭示大模型在追问场景下表现骤降

ThReadMed-QA: A Multi-Turn Medical Dialogue Benchmark from Real Patient Questions

  • 从Reddit真实问诊帖提取2437条完整对话,含8204个问答对,最多9轮
  • 最强模型GPT-5在第三轮时正确率仅41.2%,错误率比首轮翻三倍
  • 提出一致性与错误传播度量,发现错答会显著放大后续错误概率

现有医学问答基准多聚焦单轮交互,难以反映真实问诊中反复追问的特点。我们提出ThReadMed-QA,一个来自r/AskDocs的真实患者-医生对话基准,包含2,437条完整对话线程,共8,204个问答对,最多达9轮。不同于以往依赖模拟对话、对抗提示或考试式问题的研究,本数据集捕捉了患者真实的后续追问和经验证的医生回复。我们在238个对话(948个问答对)的分层测试集上评估了五款先进大模型:GPT-5、GPT-4o、Claude Haiku、Gemini 2.5 Flash 和 Llama 3.3 70B,使用基于医生真实答案校准的LLM评分标准。即使最强模型GPT-5,全正确率也仅达41.2%。所有模型从第0轮到第2轮性能显著下降(p < 0.001),第三轮错误率约为首轮的三倍。我们发现单轮能力与多轮可靠性存在根本矛盾:初始表现最强的模型(如GPT-5: 75.2;Claude Haiku: 72.3)在第二轮下降最剧烈(分别下降16.2和25.0分),而较弱模型则趋于稳定或小幅提升。我们引入两个新指标量化多轮失败模式:对话一致性得分(CCS)和错误传播率(EPR)。CCS显示近三分之一的Claude Haiku对话在同一线程内出现完全正确与完全错误响应的剧烈波动;EPR表明单次错误会使后续出错概率提高1.9–6.1倍。

原文摘要 · Abstract (English)

Medical question-answering benchmarks predominantly evaluate single-turn exchanges, failing to capture the iterative, clarification-seeking nature of real patient consultations. We introduce ThReadMed-QA, a benchmark of 2,437 fully-answered patient-physician conversation threads extracted from r/AskDocs, comprising 8,204 question-answer pairs across up to 9 turns. Unlike prior work relying on simulated dialogues, adversarial prompts, or exam-style questions, ThReadMed-QA captures authentic patient follow-up questions and verified physician responses, reflecting how patients naturally seek medical information online. We evaluate five state-of-the-art LLMs -- GPT-5, GPT-4o, Claude Haiku, Gemini 2.5 Flash, and Llama 3.3 70B -- on a stratified test split of 238 conversations (948 QA pairs) using a calibrated LLM-as-a-judge rubric grounded in physician ground truth. Even the strongest model, GPT-5, achieves only 41.2% fully-correct responses. All five models degrade significantly from turn 0 to turn 2 (p < 0.001), with wrong-answer rates roughly tripling by the third turn. We identify a fundamental tension between single-turn capability and multi-turn reliability: models with the strongest initial performance (GPT-5: 75.2; Claude Haiku: 72.3 out of 100) exhibit the steepest declines by turn 2 (dropping 16.2 and 25.0 points respectively), while weaker models plateau or marginally improve. We introduce two metrics to quantify multi-turn failure modes: Conversational Consistency Score (CCS) and Error Propagation Rate (EPR). CCS reveals that nearly one in three Claude Haiku conversations swings between a fully correct and a completely wrong response within the same thread. EPR shows that a single wrong turn raises the probability of a subsequent wrong turn by 1.9-6.1x across all models.

医学问答多轮对话大模型评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。