arXiv:2510.16194cs.AI2025-10被引 3

用多个AI代理自动评估医疗文本去标识模型,省去人工标注。

Towards Automatic Evaluation and Selection of PHI De-identification Models via Multi-Agent Collaboration

  • 多个AI代理独立判断医疗信息提取正确性,输出结构化指标。
  • 通过LLM多数投票整合结果,稳定可靠地选出最优模型。
  • 在真实临床数据上表现接近人工标注,适合资源有限场景。

保护性健康信息(PHI)去标识化对临床笔记的安全再利用至关重要,但现有评估依赖昂贵的小规模专家标注。我们提出TEAM-PHI框架,利用大语言模型(LLMs)自动衡量去标识化质量并选择最优模型,减少对真实标签的依赖。该框架部署多个评估代理,各自独立判断PHI提取的正确性,并输出结构化指标。这些结果通过基于LLM的多数投票机制整合,融合多样评估视角,生成单一、稳定且可复现的排名。在真实临床笔记语料上的实验表明,尽管个体评估者存在差异,但基于LLM的投票能可靠收敛至相同最优系统。与真实标注和人工评估的对比显示,自动化排名与监督评估高度一致。通过结合独立评估代理与LLM多数投票,TEAM-PHI为去标识化模型评估与优选提供了一种实用、安全、低成本的解决方案,尤其适用于真实标签稀缺的情况。

原文摘要 · Abstract (English)

Protected health information (PHI) de-identification is critical for enabling the safe reuse of clinical notes, yet evaluating and comparing PHI de-identification models typically depends on costly, small-scale expert annotations. We present TEAM-PHI, a multi-agent evaluation and selection framework that uses large language models (LLMs) to automatically measure de-identification quality and select the best-performing model without heavy reliance on gold labels. TEAM-PHI deploys multiple Evaluation Agents, each independently judging the correctness of PHI extractions and outputting structured metrics. Their results are then consolidated through an LLM-based majority voting mechanism that integrates diverse evaluator perspectives into a single, stable, and reproducible ranking. Experiments on a real-world clinical note corpus demonstrate that TEAM-PHI produces consistent and accurate rankings: despite variation across individual evaluators, LLM-based voting reliably converges on the same top-performing systems. Further comparison with ground-truth annotations and human evaluation confirms that the framework's automated rankings closely match supervised evaluation. By combining independent evaluation agents with LLM majority voting, TEAM-PHI offers a practical, secure, and cost-effective solution for automatic evaluation and best-model selection in PHI de-identification, even when ground-truth labels are limited.

医疗AI去标识化LLM应用自动评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。