推理步骤越多，AI在病历问答中越容易出错，且此规律跨模型一致。
Compositional Reasoning Depth Predicts Clinical AI Failure: Empirical Evidence Consistent with Transformer Compositionality Limits in Electronic Health Record Question Answering
- 用推理步数（hop count）量化临床问答复杂度，作为预测模型失败的指标。
- 三款主流大模型在四步推理时准确率降至17.6%~23.5%，随步数增加单调下降。
- 扩展思维未显著改善表现，适合关注医疗AI部署风险的临床开发者参考。
综合准确率指标掩盖了大语言模型在电子健康记录（EHR）问答中的系统性失败模式：需要更多推理步骤的问题产生显著更高错误率。受变压器模型组合性限制理论启发，我们引入预设的跳数分类法——即从EHR回答临床问题所需的不同推理步骤数——作为模型失败的合理预测因子。我们在4个跳数层级上标注了313对由临床医生生成的MedAlign EHR问答对，并评估了301个问题在三种条件下：Claude Sonnet-4-6（零样本与扩展思考）、GPT-4o 和 GPT-5.4-2026-03-05（零样本）。所有三个模型跨越两家厂商及两代OpenAI架构均显示准确率随跳数单调下降：Claude Sonnet 零样本从30.6%（跳数=1）降至17.6%（跳数=4）（Cochran-Armitage z=-2.30, p=0.011；每跳比值比0.72，95%置信区间[0.56,0.92]，p=0.008）；GPT-4o 复现该趋势（37.8% → 14.7%；比值比0.58 [0.45,0.75]，p<0.001）；GPT-5.4-2026-03-05 也确认（37.8% → 23.5%；比值比0.80 [0.66,0.98]，p=0.027）。预设上下文充分性审计显示，高跳数问题并非因病历截断而更难回答（跳数2-4时可答性为93%-95%，跳数1为79%），因此下降源于组合推理难度。扩展思考未能显著缓解准确率-深度曲线，且思考令牌使用量与跳数呈正相关（r=0.31, p<0.0001），符合预期的O(k)计算需求。跳数因此成为理论驱动、跨架构的大语言模型在EHR问答中出错的可靠预测因子，对临床AI部署风险分层具有直接意义。
原文摘要 · Abstract (English)
Aggregate accuracy benchmarks conceal a systematic structure in how large language models fail at electronic health record (EHR) question answering: questions requiring more inferential steps produce disproportionately more errors. Motivated by theoretical results on transformer compositionality limits, we introduce a pre-specified hop-count taxonomy -- the number of distinct reasoning steps required to answer a clinical question from an EHR -- as a principled predictor of model failure. We annotate 313 clinician-generated MedAlign EHR question-answer pairs across four hop levels and evaluate 301 questions in a within-model ablation (claude-sonnet-4-6, zero-shot vs. extended thinking) and cross-architecture replications (gpt-4o and gpt-5.4-2026-03-05, zero-shot). All three models, spanning two providers and two OpenAI generations (GPT-4 and GPT-5), show monotone accuracy decline with hop count: Claude Sonnet zero-shot falls from 30.6% (hop=1) to 17.6% (hop=4) (Cochran-Armitage z=-2.30, p=0.011; OR per hop 0.72, 95% CI [0.56,0.92], p=0.008); GPT-4o replicates this (37.8% to 14.7%; OR 0.58 [0.45,0.75], p<0.001); and gpt-5.4-2026-03-05 confirms it (37.8% to 23.5%; OR 0.80 [0.66,0.98], p=0.027). A pre-specified context-sufficiency audit shows higher-hop questions are not differentially disadvantaged by EHR truncation (answerability 93-95% at hops 2-4 vs. 79% at hop=1), so the decline reflects compositional reasoning difficulty. Extended thinking did not significantly flatten the accuracy-depth curve across three reasoning conditions, and thinking-token usage scaled with hop count (r=0.31, p<0.0001), consistent with the predicted O(k) computational requirement. Hop count is thus a theory-motivated, cross-architecture predictor of large-language-model error on EHR question answering, with direct implications for deployment risk stratification of clinical AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。