arXiv:2411.17967cs.CL2024-11被引 1

用大模型从社交媒体提取糖尿病药讨论中的临床数据,又快又准。

QuaLLM-Health: An Adaptation of an LLM-Based Framework for Quantitative Data Extraction from Online Health Discussions

  • 基于GPT-4o-mini,通过人工标注和提示工程优化提取流程。
  • 对2059条数据的提取准确率超0.85,平均F1超0.90,稳定匹配率达95%。
  • 成本低于3美元,一小时完成百万级数据处理,适合医疗研究应用。

Reddit等社交平台上的健康讨论蕴含宝贵信息,但从中提取定量数据极具挑战。本文将QuaLLM框架改编为QuaLLM-Health,利用大语言模型从涉及GLP-1受体激动剂的五个相关社区中提取临床相关定量数据。2024年7月通过Reddit API收集41万篇帖子与评论,剔除癌症相关内容后保留2,059条唯一条目。制定标注指南,由两名领域专家独立标注100条样本,构建金标准数据集。采用迭代提示工程,在该数据集上使用OpenAI的GPT-4o-mini训练优化管道,实现对癌症幸存、家族史、癌症类型、风险认知及医患交流等变量的提取。优化后的模型在所有变量上准确率均高于0.85,精度、召回率与F1分数宏平均值超过0.90,表现均衡;稳定性测试显示跨运行匹配率达95%。应用于全量数据后,可在约一小时内完成高效提取,成本低于3美元。结果表明,结合人类专家与提示迭代可实现大模型对非结构化社交媒体数据中临床定量信息的可靠提取,该方法可推广至多类健康领域的大规模患者生成数据研究。

原文摘要 · Abstract (English)

Health-related discussions on social media like Reddit offer valuable insights, but extracting quantitative data from unstructured text is challenging. In this work, we present an adapted framework from QuaLLM into QuaLLM-Health for extracting clinically relevant quantitative data from Reddit discussions about glucagon-like peptide-1 (GLP-1) receptor agonists using large language models (LLMs). We collected 410k posts and comments from five GLP-1-related communities using the Reddit API in July 2024. After filtering for cancer-related discussions, 2,059 unique entries remained. We developed annotation guidelines to manually extract variables such as cancer survivorship, family cancer history, cancer types mentioned, risk perceptions, and discussions with physicians. Two domain-experts independently annotated a random sample of 100 entries to create a gold-standard dataset. We then employed iterative prompt engineering with OpenAI's "GPT-4o-mini" on the gold-standard dataset to build an optimized pipeline that allowed us to extract variables from the large dataset. The optimized LLM achieved accuracies above 0.85 for all variables, with precision, recall and F1 score macro averaged > 0.90, indicating balanced performance. Stability testing showed a 95% match rate across runs, confirming consistency. Applying the framework to the full dataset enabled efficient extraction of variables necessary for downstream analysis, costing under $3 and completing in approximately one hour. QuaLLM-Health demonstrates that LLMs can effectively and efficiently extract clinically relevant quantitative data from unstructured social media content. Incorporating human expertise and iterative prompt refinement ensures accuracy and reliability. This methodology can be adapted for large-scale analysis of patient-generated data across various health domains, facilitating valuable insights for healthcare research.

大模型健康数据量化提取社交媒体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。