arXiv:2608.14683cs.LG2026-08

诊断系统如何在罕见病中可靠选择预测结果,关键在于分数与排名的匹配。

One Score, Two Decisions: Selective Prediction on the Rare-Disease Tail

论文配图:One Score, Two Decisions: Selective Prediction on the Rare-Disease Tail
图 1 · 摘自论文原文
  • 用排名和置信度双重判断是否采纳首推诊断
  • 超罕见病召回率仅4.6%,10%覆盖下准确率难超50%
  • 顶二分差比最高分更能反映诊断可信度

面对患者临床表现,诊断系统需对疾病排名并决定何时采纳首个预测或推迟审查。传统做法是阈值化最高得分。选择性预测需满足两个条件:其一,排序器必须具备足够高的一级召回能力以实现目标可行性;在按疾病流行度分层的2,000例患者记录中,八种小型开放权重LLM在极罕见病上的Recall@1最高仅为4.6%。即使信心评分完美,10%覆盖率下也难以达到50%的选择性准确率。更优模型同样受限,说明此极限属于特定场景。其二,置信信号必须与决策任务匹配;对于固定候选集的排序器,顶二分差会消除跨候选项共享成分。在仅基于表型的Exomiser上,该指标使10%病例的准确率达29.0%,而整体准确率为13.3%;相比之下,最高分无法提供有效筛选。然而,这种抵消也可能抹除识别候选列表中是否存在正确答案所需的信息。SciFact检索与生物医学实体链接验证了这一区别。最后,我们证明仅凭未标注得分无法判断切换至分差是否会带来改进。

原文摘要 · Abstract (English)

Given a patient's clinical findings, a diagnostic system ranks possible diseases and must decide when to endorse its first prediction or defer it for review. This decision is usually made by thresholding the top score. Selective prediction over ranked outputs begins with two checks. First, the ranker must produce enough correct top-ranked predictions to make the target feasible. Across 2,000 patient records stratified by disease prevalence, eight small open-weight LLMs achieve at most 4.6% Recall@1 on ultra-rare diseases. At 10% coverage, even a perfect confidence ranking of their existing predictions therefore cannot reach 50% selective accuracy. More accurate models pass the same check, showing that the limit is regime-specific. Second, the confidence signal must match the decision being made. For fixed-candidate rankers, the top-two margin cancels components shared across candidates. On phenotype-only Exomiser, it selects 10% of cases at 29.0% accuracy, compared with 13.3% overall, while the top score provides no reliable gate. Yet that cancellation can remove information needed to detect whether the candidate list contains an answer. SciFact retrieval and biomedical entity linking confirm this distinction. Finally, we prove that unlabelled scores alone cannot determine whether switching to the margin will help.

诊断系统罕见病选择性预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。