arXiv:2608.16643cs.CLcs.AI2026-08中稿 · Machine Learning f…

现有临床错误检测评估方法易误判模型能力,需改用成对比较。

Toward Better Assessment of LLMs' Performance in Clinical Error Detection

论文配图:Toward Better Assessment of LLMs' Performance in Clinical Error Detection
图 1 · 摘自论文原文
  • 通过成对对比病历判断模型真实辨识能力
  • 15个模型中13个连随机猜都比不上,但F1分数看似中等
  • 不同语言下模型倾向不同,建议补充成对评估

大型语言模型在临床文档错误检测中具有应用前景,但当前评估多孤立分析每份病历。基准测试通常通过向病历注入错误生成配对样本,但聚合指标(如平衡准确率或F1)未利用此结构。我们发现该忽略后果严重:在跨3种语言的4个标准化测试集上评估15个多样化的LLM时,13个模型的成对辨别能力低于随机水平,尽管其F1分数被常规视为中等。同时观察到不同语言下模型存在不同偏差——同一模型在一种语言中倾向于忽略错误,在另一种中则过度报警。为诊断失败原因,我们引入评分机制以评估模型输出所依据的证据。结果发现,模型虽能定位错误相关内容,却无法对无错对照文本给出正确判断。进一步显示,F1与成对准确率受相同底层偏差驱动,方向相反,因此按F1排序可能系统性地推广最弱的辨别器。对于安全关键的临床NLP应用,我们建议在基准报告中补充成对评估。代码与分析脚本见https://github.com/healthylaife/paired-clinical-eval。

原文摘要 · Abstract (English)

Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation. Error-detection benchmarks are typically constructed by injecting errors into notes, such that each erroneous note has a natural counterpart. Aggregate discriminative metrics (e.g., balanced accuracy or F1) do not exploit this structure. We show that this omission is consequential. In particular, evaluating 15 diverse LLMs on 4 standardized clinical error-detection test sets across 3 languages, we find that 13 of 15 models fall below the level of random pairwise discrimination, even while achieving F1 scores that standard practice would read as moderate. We also observe that the underlying bias patterns differ across languages: the same model can default to "no error" on one language and over-flag errors on another. To diagnose where discrimination breaks down, we further introduce a procedure to score the evidence models cite in their outputs. We find that while models consistently locate error-relevant content, they fail to produce the corresponding correct verdict on the clean counterpart. Finally, we show that F1 and pairwise accuracy are driven in opposite directions by the same underlying bias, so that ranking models by F1 may systematically promote the weakest discriminators. For safety-critical clinical NLP applications, we advocate for supplementing aggregate metrics with paired evaluations in benchmark reporting. Code and analysis scripts are available at https://github.com/healthylaife/paired-clinical-eval.

临床NLP大模型评估错误检测成对对比

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。