测试临床问答中检索的准确性，发现真实病历中的否定、时间等信息极易导致错误。
ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV
- 用带标注的医学知识图谱引导检索，按问题意图匹配答案
- 新方法比基线提升22个百分点，尤其在正确理解问题意图时效果更佳
- 适合关注临床问答系统真实场景性能的研究者与开发者
现有推理基准多在干净输入上评估临床表现。本文聚焦推理前的关键步骤——在真实电子病历中进行检索，发现否定、时间关系及家属与患者归属关系的误判可能导致正确答案变为错误。EpiKG为每个事实附带断言标签和时间标记，并按问题意图引导检索。ClinicalBench是一个覆盖43名MIMIC-IV患者的400道题测试集，涵盖9类敏感于断言的类别。通过7条件消融实验，在6个大模型（Claude Opus 4.6, GPT-OSS 20B, MedGemma 27B, Gemma 4 31B, MedGemma 1.5 4B, Qwen 3.5 35B）上验证。三位医生盲评100对样本。以作者盲法为主终点，采用留作者外配对精确麦克内马尔检验（50个一致严格项，由两名外部医生评分），结果显示提升22.0个百分点（95%置信区间[+5.1, +31.5]，p=0.0192）。架构创新——基于意图感知的知识图谱检索增强生成（KG-RAG）相比对比基线Contriever dense-RAG，在排除变化项的n=362终点上提升8.84个百分点（配对麦克内马尔p=1.79e-3）；在理想意图条件下提升12.43个百分点。敏感性分析方向一致：三名医生多数意见下提升24.0个百分点（受单作者循环性影响）；确定性关键词复现代理指标提升39.5个百分点。六模型整体趋势显示，随着大模型独立表现上升，增益随模型规模增长而缩小（beta=-1.123, r=-0.921, p=0.009），更似回归均值而非模型大小替代编码。医生评审发现56%自动生成参考答案存在缺陷，提示临床问答基准必须经医生审定才可用。本研究公开了ClinicalBench评测集、三名医生评审数据及EpiKG输出结果。
原文摘要 · Abstract (English)
Reasoning benchmarks measure clinical performance on clean inputs. We evaluate the step before reasoning: retrieval over real EHR notes, where negation, temporality, and family-versus-patient attribution can flip a correct answer to a wrong one. EpiKG carries an assertion label and a temporality tag with every fact in a patient knowledge graph, then routes retrieval by question intent. ClinicalBench is a 400-question test over 43 MIMIC-IV patients across 9 assertion-sensitive categories. A 7-condition ablation tests each piece of EpiKG across six LLMs (Claude Opus 4.6, GPT-OSS 20B, MedGemma 27B, Gemma 4 31B, MedGemma 1.5 4B, Qwen 3.5 35B). Three physicians blindly adjudicated 100 paired items. The author-blind primary endpoint, leave-author-out paired exact McNemar on 50 unanimous-strict items rated by two external physicians, yields +22.0 percentage points (95 percent Newcombe CI [+5.1, +31.5], p=0.0192). The architectural novelty, intent-aware KG-RAG over a Contriever dense-RAG baseline (C2b to C4g_kw on the change-excluded n=362 endpoint), is +8.84 percentage points (paired McNemar p=1.79e-3); +12.43 percentage points under oracle intent. Sensitivities agree directionally: three-rater physician majority +24.0 percentage points (subject to single-author circularity); deterministic keyword reproducibility proxy +39.5 percentage points. Across the six models, the gain shrinks as the LLM-alone baseline rises (beta=-1.123, r=-0.921, p=0.009). With n=6 this looks more like regression to the mean than encoding substituting for model size. Physician adjudication identified 56 percent of auto-generated reference answers as defective, a methodological finding indicating that NLP-pipeline clinical-QA benchmarks require physician adjudication to be usable. ClinicalBench, the frozen evaluator, three-rater adjudication data, and the EpiKG output stack are publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。