用大模型分析住院患者评论,自动分类多标签反馈。
Large Language Models for Patient Comments Multi-Label Classification
- 用GPT-4 Turbo结合提示工程实现零样本/少样本多标签分类。
- 在真实数据上达到76.12%的F1分数,优于传统方法。
- 内置隐私保护机制,适合医疗健康领域应用。
患者体验与医疗质量对医院可持续发展和声誉至关重要。分析患者反馈可提供满意度与治疗结果的重要洞察,但其非结构化特性使传统监督学习面临标注数据稀缺与文本细微差异的挑战。本研究探索利用大语言模型(LLMs)对住院患者评论进行多标签文本分类(MLTC)。采用GPT-4 Turbo完成分类任务,并引入受保护健康信息(PHI)检测框架,在输入前确保患者身份脱敏。通过提示工程,实验了零样本学习、上下文学习及思维链提示。结果显示,无论零样本或少样本设置,GPT-4 Turbo均优于传统方法与预训练语言模型(PLMs),取得76.12%的整体F1分数与73.61%加权F1分数,少样本结果紧随其后。进一步分析了分类结果与其他患者体验结构化变量(如评分)的相关性。该研究通过大模型提升了多标签分类性能,为医疗从业者高效理解患者反馈、及时响应提供支持。
原文摘要 · Abstract (English)
Patient experience and care quality are crucial for a hospital's sustainability and reputation. The analysis of patient feedback offers valuable insight into patient satisfaction and outcomes. However, the unstructured nature of these comments poses challenges for traditional machine learning methods following a supervised learning paradigm. This is due to the unavailability of labeled data and the nuances these texts encompass. This research explores leveraging Large Language Models (LLMs) in conducting Multi-label Text Classification (MLTC) of inpatient comments shared after a stay in the hospital. GPT-4 Turbo was leveraged to conduct the classification. However, given the sensitive nature of patients' comments, a security layer is introduced before feeding the data to the LLM through a Protected Health Information (PHI) detection framework, which ensures patients' de-identification. Additionally, using the prompt engineering framework, zero-shot learning, in-context learning, and chain-of-thought prompting were experimented with. Results demonstrate that GPT-4 Turbo, whether following a zero-shot or few-shot setting, outperforms traditional methods and Pre-trained Language Models (PLMs) and achieves the highest overall performance with an F1-score of 76.12% and a weighted F1-score of 73.61% followed closely by the few-shot learning results. Subsequently, the results' association with other patient experience structured variables (e.g., rating) was conducted. The study enhances MLTC through the application of LLMs, offering healthcare practitioners an efficient method to gain deeper insights into patient feedback and deliver prompt, appropriate responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。