arXiv:2606.23992cs.CLcs.AI2026-06

用检索约束的LLM提升临床术语集生成准确率

RASC+: Retrieval-Constrained LLM Adjudication for Clinical Value Set Authoring

  • 先高召回检索候选代码,再用受限LLM筛选最优组合
  • 在3744个术语集上,宏观F1从0.287提升至0.549
  • 适合需要高安全性和可审计性的医疗AI研发人员

临床价值集定义了质量测量、表型识别、队列构建和临床决策支持中使用的标准化术语代码。近期提出的检索增强集合补全(RASC)基准表明,直接零样本大语言模型生成不适用于此任务:临床代码体系庞大、版本受控,且无法被语言模型可靠记忆。本文研究了一种分阶段方法,其中候选池构建优化召回率,受限的LLM评判器优化候选选择。在完整的3,744个价值集测试集上,基于Qwen3的词汇感知扩展与代码显示恢复检索,将候选池召回率从原始RASC基线0.553提升至0.730;在保留发布者子集上,召回率为0.655。仅提高召回率仍不足:使用原始SAPBert交叉编码器在此扩展池上,全测试集宏观F1为0.287,保留发布者子集为0.233。将第二阶段选择器替换为盲视GPT-5评判,相同池上全测试集宏观F1提升至0.549,保留发布者子集为0.533。结果表明,检索约束的LLM评判可显著提升价值集补全性能,同时保持所有返回代码均来自可审计候选池的安全约束。

原文摘要 · Abstract (English)

Clinical value sets define the standardized terminology codes used in quality measurement, phenotyping, cohort construction, and clinical decision support. The recently introduced Retrieval-Augmented Set Completion (RASC) benchmark showed that direct zero-shot large language model (LLM) generation is poorly suited to this task: clinical code systems are large, version-controlled, and not reliably memorized by language models. We study a stage-wise alternative in which candidate-pool construction is optimized for recall and a constrained LLM adjudicator is optimized for candidate selection. On the full 3,744-value-set RASC test split, Qwen3-based retrieval with vocabulary-aware expansion and code-display rescue retrieval increases candidate-pool recall from the original RASC retrieval baseline of 0.553 to 0.730; on the held-out-publisher stratum, pool recall is 0.655. The higher-recall pool alone is not sufficient: applying the original SAPBert cross-encoder to this expanded pool gives full-test macro F1 of 0.287 and held-out-publisher macro F1 of 0.233. Replacing the stage-2 selector with blinded GPT-5 adjudication over the same pool increases full-test macro F1 to 0.549 and held-out-publisher macro F1 to 0.533. These results show that retrieval-constrained LLM adjudication can substantially improve value set completion while preserving the safety constraint that all returned codes must come from an auditable candidate pool.

医疗AI术语生成LLM评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。