超四成生物医学论文用的AI模型已停用或即将停用,影响研究可复现性。
Model Retirement Creates Reproducibility Risk in Biomedical AI Publications
- 分析50个常用大模型在生物医学论文中的使用情况
- 42%论文引用的模型已在发表时停用或两年内将停用
- 模型退役周期中位数仅538天,急需建立报告规范
大型语言模型在生物医学研究中应用迅速增长,但许多广泛使用的模型由商业服务提供,存在明确的淘汰时间表,可能威胁科学可复现性。我们通过PubMed检索2022年至2026年3月期间使用特定大模型进行生物医学任务的原始研究论文,从61,077篇摘要中提取模型名称,并由人工审核验证准确性。提取的模型名称被归一化为标准标识符。整理了前50个最常用模型的生命周期数据(发布日期、退役日期、状态)。共识别出8,931条论文-模型提及,涵盖5,242篇独立论文。其中77.7%引用的是商业闭源模型。总体而言,42%的引用涉及在正式发表时已退役或计划在发表后两年内退役的模型。论文发表至模型退役的中位时间为538天。结论:大量使用大模型的生物医学研究正面临发表后计算不可复现的风险。模型淘汰应被视为生物医学研究中核心的报告与保存问题。
原文摘要 · Abstract (English)
Background. Large language models (LLMs) are being adopted in biomedical research at a rapid and accelerating pace, yet commercial services that host many widely used models operate under deprecation schedules that can complicate scientific reproducibility. Methods. We searched PubMed for original research articles from 2022 through March 2026 that applied a specific LLM to a biomedical task. An extraction agent identified model names from 61,077 article abstracts with human reviewers validating a subset for extraction accuracy. Extracted model names were normalized to canonical model identifiers. Lifecycle data (release date, retirement date, status) were compiled for the 50 most frequently used models. Results. We identified 8,931 paper-model mentions spanning 5,242 unique publications after restricting the analysis to the 50 most frequently used models. Among these mentions, 77.7% cited a commercial closed-weight model. Overall, 42% involved a model that was already retired by the time of official publication or is scheduled to retire within two years of publication. The median interval from publication to model retirement was 538 days. Conclusion. Many biomedical publications using LLMs are on a trajectory toward computational non-reproducibility after publication. Model deprecation should be treated as a core reporting and preservation issue for biomedical research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。