量化数据集多样性对模型性能的影响，发现扫描仪是主要多样性来源但会引发捷径学习。
Dataset Diversity Metrics and Impact on Classification Models
- 用可控扰动的MorphoMNIST和真实胸片数据集PadChest评估多种多样性度量方法。
- 图像与元数据的无参考度量与模型AUC相关性弱，而FID和语义多样性度量相关性强。
- 临床专家认为扫描仪是主要多样性来源，但新增扫描仪反而导致模型捷径学习。
训练数据集的多样性通常被视为获得稳健模型的关键因素，但其定义常不明确或因论文而异。尽管已有部分度量方法，但在开发新算法时，多样性量化往往被忽视。本文在具有可控扰动的MorphoMNIST和公开胸片数据集PadChest上，研究了图像、文本和元数据的多种数据集多样性度量行为。评估这些度量彼此间的相关性，以及与临床专家直觉的相关性，并检验其与下游任务性能及模型训练动态的关系。结果表明，图像或元数据的无参考多样性度量与模型AUC相关性有限，而与FID及语义多样性度量相关性较高。临床专家指出扫描仪是实践中主要的多样性来源，但实验发现增加另一个扫描仪至训练集会导致模型出现捷径学习现象。研究代码已公开于https://github.com/TheoSourget/dataset_diversity_evaluation。
原文摘要 · Abstract (English)
The diversity of training datasets is usually perceived as an important aspect to obtain a robust model. However, the definition of diversity is often not defined or differs across papers, and while some metrics exist, the quantification of this diversity is often overlooked when developing new algorithms. In this work, we study the behaviour of multiple dataset diversity metrics for image, text and metadata using MorphoMNIST, a toy dataset with controlled perturbations, and PadChest, a publicly available chest X-ray dataset. We evaluate whether these metrics correlate with each other but also with the intuition of a clinical expert. We also assess whether they correlate with downstream-task performance and how they impact the training dynamic of the models. We find limited correlations between the AUC and image or metadata reference-free diversity metrics, but higher correlations with the FID and the semantic diversity metrics. Finally, the clinical expert indicates that scanners are the main source of diversity in practice. However, we find that the addition of another scanner to the training set leads to shortcut learning. The code used in this study is available at https://github.com/TheoSourget/dataset_diversity_evaluation
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。