用贝叶斯方法衡量数据样本有用性，可高效筛选关键数据。
Laplace Sample Information: Data Informativeness Through a Bayesian Lens
- 基于贝叶斯近似与KL散度，量化单个样本对模型参数的影响。
- 在图像和文本数据上有效识别异常样本、判断类别重要性与数据难度。
- 计算轻量且可迁移，适合大规模模型训练中的数据筛选。
准确估计数据集中单个样本的有用性是深度学习中的重要目标，有助于指导样本选择，通过剔除冗余或有害样本提升模型效率与精度。本文提出一种基于信息论的通用样本有用性度量方法——拉普拉斯样本信息（Laplace Sample Information, LSI），该方法利用贝叶斯对权重后验的近似以及KL散度，衡量某一样本引入前后参数分布的变化。实验表明，LSI在排序数据典型性、检测误标样本、衡量类别层面信息量及评估数据集难度方面均表现良好。我们在监督与非监督设置下的图像与文本数据上验证了其有效性。此外，通过探针机制可高效计算LSI，且能良好迁移至大模型训练中。
原文摘要 · Abstract (English)
Accurately estimating the informativeness of individual samples in a dataset is an important objective in deep learning, as it can guide sample selection, which can improve model efficiency and accuracy by removing redundant or potentially harmful samples. We propose Laplace Sample Information (LSI) measure of sample informativeness grounded in information theory widely applicable across model architectures and learning settings. LSI leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by a sample of interest from the dataset. We experimentally show that LSI is effective in ordering the data with respect to typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty. We demonstrate these capabilities of LSI on image and text data in supervised and unsupervised settings. Moreover, we show that LSI can be computed efficiently through probes and transfers well to the training of large models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。