74%的医疗AI论文不公开数据或代码,开源能显著提升可复现性与影响力。
Bridging the Reproducibility Divide: Open Source Software's Role in Standardizing Healthcare AI
- 推动使用公开数据和共享代码以统一医疗AI研究标准。
- 开源论文平均被引次数比非开源多110%,影响更大。
- 适合关注医疗AI可信度与落地的研究者和从业者。
对近期医疗AI(AI4H)论文的分析显示,尽管存在使用公开数据集和共享代码的趋势,仍有74%的论文依赖私有数据或未公开代码,这在强调信任的医疗领域尤为严重。不一致且文档缺失的数据预处理流程导致相同任务和数据下模型表现差异大,难以评估真实效果。尽管可复现性要求增加研究负担,但带来显著收益:使用公共数据并共享代码的论文平均被引量比两者均未用的高出110%以上。因此,医疗AI社区亟需推广开放科学实践,制定标准化数据预处理规范,建立稳健基准。通过开源开发解决这些问题,可提升可复现性,确保AI模型安全、有效,促进其在临床中的应用,最终改善患者结果并推动医学进步。
原文摘要 · Abstract (English)
Our analysis of recent AI4H publications reveals that, despite a trend toward utilizing open datasets and sharing modeling code, 74% of AI4H papers still rely on private datasets or do not share their code. This is especially concerning in healthcare applications, where trust is essential. Furthermore, inconsistent and poorly documented data preprocessing pipelines result in variable model performance reports, even for identical tasks and datasets, making it challenging to evaluate the true effectiveness of AI models. Despite the challenges posed by the reproducibility crisis, addressing these issues through open practices offers substantial benefits. For instance, while the reproducibility mandate adds extra effort to research and publication, it significantly enhances the impact of the work. Our analysis shows that papers that used both public datasets and shared code received, on average, 110% more citations than those that do neither--more than doubling the citation count. Given the clear benefits of enhancing reproducibility, it is imperative for the AI4H community to take concrete steps to overcome existing barriers. The community should promote open science practices, establish standardized guidelines for data preprocessing, and develop robust benchmarks. Tackling these challenges through open-source development can improve reproducibility, which is essential for ensuring that AI models are safe, effective, and beneficial for patient care. This approach will help build more trustworthy AI systems that can be integrated into healthcare settings, ultimately contributing to better patient outcomes and advancing the field of medicine.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。