用微调取代提示词,让模型更准识别临床记录中的偏见语言。
Fine-Tune, Don't Prompt, Your Language Model to Identify Biased Language in Clinical Notes
- 用带情绪评分的术语词典提取病历片段,再通过微调模型分类偏见
- 微调后模型在妇产科数据上F1达0.96,显著优于提示工程方法
- 不同专科中同一词含义可能不同,需按专科定制训练
临床记录中常包含带有情绪色彩的歧视性或特权化表述。本文构建了一个经过筛选的情绪评分术语词典,用于检测和分类此类语言。通过词典匹配从纽约西奈山医院妇产科分娩记录和多专科MIMIC-IV出院摘要中提取文本片段,由三位临床医生标注,分析不同专科与医疗系统中的情绪模式。对比了零样本提示、上下文学习与监督微调等多种分类策略,涵盖编码器模型(GatorTron)与生成式大模型(Llama)。结果显示,使用词典引导输入的微调方法始终优于提示方法。GatorTron在妇产科测试集上达到F1=0.96,性能超越更大生成模型,且所需提示工程少、计算资源低。在MIMIC-IV上外部验证显示跨领域泛化能力有限(F1<0.70,下降44%)。在更广义的MIMIC-IV数据上训练后,测试回妇产科的F1为0.71(下降11%),但精度下降。研究证明,微调优于提示,且模型必须针对具体医学专科进行适配才能获得临床可用性能。同一术语在不同语境下可能具有不同情绪价值——临床意义的词在另一场景中可能构成歧视。因此,在偏见检测中,误判可能损害医患信任或造成患者伤害,必须采用专科特异性微调以捕捉语义变化。
原文摘要 · Abstract (English)
Clinical documentation can contain emotionally charged language with stigmatizing or privileging valences. We present a framework for detecting and classifying such language as stigmatizing, privileging, or neutral. We constructed a curated lexicon of biased terms scored for emotional valence. We then used lexicon-based matching to extract text chunks from OB-GYN delivery notes (Mount Sinai Hospital, NY) and MIMIC-IV discharge summaries across multiple specialties. Three clinicians annotated all chunks, enabling characterization of valence patterns across specialties and healthcare systems. We benchmarked multiple classification strategies (zero-shot prompting, in-context learning, and supervised fine-tuning) across encoder-only models (GatorTron) and generative large language models (Llama). Fine-tuning with lexically primed inputs consistently outperformed prompting approaches. GatorTron achieved an F1 score of 0.96 on the OB-GYN test set, outperforming larger generative models while requiring minimal prompt engineering and fewer computational resources. External validation on MIMIC-IV revealed limited cross-domain generalizability (F1 < 0.70, 44% drop). Training on the broader MIMIC-IV dataset improved generalizability when testing on OB-GYN (F1 = 0.71, 11% drop), but at the cost of reduced precision. Our findings demonstrate that fine-tuning outperforms prompting for emotional valence classification and that models must be adapted to specific medical specialties to achieve clinically appropriate performance. The same terms can carry different emotional valences across specialties: words with clinical meaning in one context may be stigmatizing in another. For bias detection, where misclassification risks undermining clinician trust or perpetuating patient harm, specialty-specific fine-tuning is essential to capture these semantic shifts. * Equal contribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。