医学影像AI竞赛数据缺乏公平性,影响临床实用性。
Medical Imaging AI Competitions Lack Fairness
- 分析249项竞赛、458个任务,评估数据多样性与可获取性。
- 超半数数据集地域、模态、任务类型覆盖不足,代表性差。
- 多数数据访问受限、授权模糊,难以复现或长期使用。
基准竞赛在医学影像人工智能发展中起核心作用,定义性能标准并推动方法进步。然而,这些基准是否具备足够代表性的数据、可访问性和可重用性,以支持具有临床意义的AI仍不明确。本文通过大规模系统研究249项生物医学图像分析竞赛,涵盖458个任务和19种成像模态,评估了两个互补维度的公平性:(1)挑战数据集是否涵盖真实世界临床数据的多样性;(2)是否符合FAIR原则,即数据可访问、可合法重用。结果表明,现有挑战数据集在地理分布、成像模态和问题类型上均存在显著代表性不足,引发对当前基准是否反映真实临床多样性的担忧。尽管影响广泛,大多数数据集受限于严格或模糊的访问条件、不一致或不符合规范的许可实践,以及文档不全,严重制约了可复现性和长期重用。这些缺陷暴露了基准生态系统的根本性公平局限,并揭示了排行榜成绩与临床相关性之间的脱节。
原文摘要 · Abstract (English)
Benchmarking competitions are central to the development of artificial intelligence (AI) in medical imaging, defining performance standards and shaping methodological progress. However, it remains unclear whether these benchmarks provide data that are sufficiently representative, accessible, and reusable to support clinically meaningful AI. In this work, we assess fairness along two complementary dimensions: (1) whether challenge datasets capture the diversity of real-world clinical data, and (2) whether they are accessible and legally reusable in line with the FAIR principles. To address this question, we conducted a large-scale systematic study of 249 biomedical image analysis challenges comprising 458 tasks across 19 imaging modalities. Our findings reveal limited representation across challenge datasets with respect to geographic location, imaging modalities, and problem types, raising concerns about how well current benchmarks reflect real-world clinical diversity. Despite their widespread influence, challenge datasets were frequently constrained by restrictive or ambiguous access conditions, inconsistent or non-compliant licensing practices, and incomplete documentation, limiting reproducibility and long-term reuse. Together, these shortcomings expose foundational fairness limitations in our benchmarking ecosystem and highlight a disconnect between leaderboard success and clinical relevance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。