无需微调的多智能体系统自动提取临床症状,准确率高且可本地部署。
A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation Study
- 用多智能体自主生成优化抽取提示,不依赖人工设计或模型微调。
- 平均敏感度0.76、特异度0.95,对多数症状优于传统词典方法。
- 在低频症状上仍保持良好特异性,适合临床本地化部署场景。
临床笔记包含大量就诊原因相关的体征和症状,但这些信息极少进入结构化字段。现有提取方法要么依赖上下文无关规则导致误报,要么依赖需大量微调的监督模型。我们提出Pythia——一个无需人工提示工程或微调的多智能体系统,能自主编写并优化临床概念的抽取提示。系统运行于本地开源权重模型上,确保临床数据不外泄,并基于开发集的敏感度与特异度选择最优提示。我们在400份临床笔记(387名患者)中对比了Pythia与精选词典,涵盖72个体征症状。开发集(n=300)与验证集(n=100)按概念独立划分。Pythia平均敏感度为0.76,特异度0.95;词典分别为0.82和0.76;在62个可比概念中,Pythia在20个上表现更优或相当。对于词典将所有记录标记为正的14个概念,Pythia通过要求现时态、患者归属的表述,实现平均特异度0.97。特异度从开发到验证迁移性能稳定,而敏感度在流行率低于5%时下降,低于2%时平均差距达0.25。同开发集上微调的BERT分类器平均敏感度仅0.23,且在流行率低于5%时归零。结果表明,自主、免微调的提示优化可在开发到验证间有效泛化,同时支持本地部署。
原文摘要 · Abstract (English)
Clinical notes contain many of the signs and symptoms that bring patients to care, yet this information rarely reaches structured fields. Existing extraction approaches either rely on context-insensitive rules that generate false positives or on supervised models that require substantial fine-tuning. We present Pythia, a multi-agent system that autonomously writes and optimizes extraction prompts for clinical concepts without manual prompt engineering or fine-tuning. Running on a locally hosted open-weights model, Pythia keeps clinical notes on local infrastructure and selects prompts using development-set sensitivity and specificity. We compared Pythia with a curated lexicon across 72 signs and symptoms from 400 clinical notes representing 387 patients. Development (n=300) and validation (n=100) sets were partitioned independently for each concept. Pythia achieved mean sensitivity of 0.76 and specificity of 0.95, compared with 0.82 and 0.76 for the lexicon, and matched or exceeded the lexicon on both metrics for 20 of 62 directly comparable concepts. For 14 concepts where the lexicon labeled every note positive, Pythia recovered mean specificity of 0.97 by requiring a present-tense, patient-attributed finding rather than any textual mention of a term. Specificity transferred from development to validation with minimal degradation across prevalences, whereas sensitivity transfer weakened below 5% prevalence, reaching a mean gap of 0.25 below 2% prevalence. A BERT classifier fine-tuned per concept on the same development set achieved mean sensitivity of 0.23 and collapsed to zero sensitivity for concepts below roughly 5% prevalence. These findings suggest that autonomous, fine-tuning-free prompt optimization can produce symptom extraction prompts that generalize effectively from development to validation while remaining deployable on local infrastructure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。