用主观逻辑评估数据集偏见，支持不确定环境下全局可信度分析
Assessing Trustworthiness of AI Training Dataset using Subjective Logic -- A Use Case on Bias
- 基于主观逻辑构建数据集可信度评估框架
- 在交通标志识别数据集上验证了对类别不平衡的捕捉能力
- 适用于分布式、证据冲突等复杂场景，适合关注数据质量的研究者
随着人工智能系统日益依赖训练数据，评估数据集的可信度变得至关重要，尤其对于公平性或偏见这类仅在数据集层面显现的特性。已有研究使用主观逻辑评估单个数据的可信度，但尚未针对仅在数据集整体层面出现的可信度属性进行评估。本文首次提出一个正式框架，用于评估人工智能训练数据集的可信度，支持对偏见等全局性质的不确定性感知评估。该方法基于主观逻辑，可处理证据不完整、分布或冲突的场景，并支持信任命题的量化。我们在偏见这一可信度属性上实例化该框架，并基于交通标志识别数据集进行实验验证。结果表明，该方法能有效捕捉类别不平衡问题，在集中式和联邦学习场景下均保持可解释性和鲁棒性。
原文摘要 · Abstract (English)
As AI systems increasingly rely on training data, assessing dataset trustworthiness has become critical, particularly for properties like fairness or bias that emerge at the dataset level. Prior work has used Subjective Logic to assess trustworthiness of individual data, but not to evaluate trustworthiness properties that emerge only at the level of the dataset as a whole. This paper introduces the first formal framework for assessing the trustworthiness of AI training datasets, enabling uncertainty-aware evaluations of global properties such as bias. Built on Subjective Logic, our approach supports trust propositions and quantifies uncertainty in scenarios where evidence is incomplete, distributed, and/or conflicting. We instantiate this framework on the trustworthiness property of bias, and we experimentally evaluate it based on a traffic sign recognition dataset. The results demonstrate that our method captures class imbalance and remains interpretable and robust in both centralized and federated contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。