分析ChatGPT健康回答的引用来源,提出四维度可信度评估框架。
Authority Signals in AI Cited Health Sources: A Framework for Evaluating Source Credibility in ChatGPT Responses
- 构建四域框架:作者资质、机构背景、质量审核、数字权威。
- 75%以上引用来自梅奥诊所等权威机构,其余多为非正规渠道。
- 适合关注AI医疗信息可信度的研究者与公众用户。
大型语言模型(LLM)兴起后,健康信息获取方式发生根本改变,近三分之一的ChatGPT用户每周询问健康问题。随着医疗机构积极优化在LLM系统中的可见性,理解生成回答的来源至关重要。本研究提出权威信号框架,涵盖四个维度:谁撰写了内容(作者资质)、谁发布了内容(机构背景)、如何保证质量(质量保障)、AI如何发现它(数字权威)。研究从包含3,173个用户健康问题的HealthSearchQA中随机抽取100个问题,输入ChatGPT 5.2 Pro,记录并依据该框架编码615条引用来源。结果显示,超75%的引用来自梅奥诊所、克利夫兰诊所、维基百科、英国国家医疗服务体系、PubMed等成熟机构;剩余来源多为缺乏权威背书的替代性健康信息。
原文摘要 · Abstract (English)
Health information seeking has fundamentally changed since the onset of Large Language Models (LLM), with nearly one third of ChatGPT's 800 million users asking health questions weekly. Understanding the sources of those AI generated responses is vital, as health organizations and providers are also investing in digital strategies to organically improve their ranking, reach and visibility in LLM systems like ChatGPT. As AI search optimization strategies are gaining maturity, this study introduces an Authority Signals Framework, organized in four domains that reflect key components to health information seeking, starting with "Who wrote it?" (Author Credentials), followed by "Who published it?" (Institutional Affiliation), "How was it vetted?" (Quality Assurance), and "How does AI find it?" (Digital Authority). This descriptive cross-sectional study randomly selected 100 questions from HealthSearchQA which contains 3,173 consumer health questions curated by Google Research from publicly available search engine suggestions. Those questions were entered into ChatGPT 5.2 Pro to record and code the cited sources through the lens of the Authority Signals Framework's four domains. Descriptive statistics were calculated for all cited sources (n=615), and cross tabulations were conducted to examine distinction among organization types. Over 75% of the sources cited in ChatGPT's health generated responses were from established institutional sources, such as Mayo Clinic, Cleveland Clinic, Wikipedia, National Health Service, PubMed with the remaining citations sourced from alternative health information sources that lacked established institutional backing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。