提出量化分类标注模糊性的新方法,区分不确定来源。
Quantifying Ambiguity in Categorical Annotations: A Measure and Statistical Inference Framework
- 设计可映射响应分布为0~1标量的模糊性度量
- 能分离类别不可区分与明确无法解答的不确定性
- 提供统计推断工具,适用于数据质量评估
人类生成的分类标注常产生反映模糊性的经验响应分布(软标签),而非简单标注错误。本文提出一种模糊性度量,将离散响应分布映射到[0,1]区间内的标量,用于量化分类任务中的随机不确定性。该度量与二次熵(类似基尼不纯度)密切相关,但通过不对称处理“无法解决”类别,将因类别间难以区分产生的不确定性与因明确不可解产生的不确定性区分开来。我们分析了该度量的形式性质,并与文献中代表性模糊性度量进行对比。超越描述,本文开发了统计推断工具:提出频率学派的总体模糊性点估计器,并基于狄利克雷先验推导出模糊性上的贝叶斯后验分布,为认知不确定性提供了严谨建模。数值示例展示了估计、校准及在数据集质量评估和下游机器学习流程中的实际应用。
原文摘要 · Abstract (English)
Human-generated categorical annotations frequently produce empirical response distributions (soft labels) that reflect ambiguity rather than simple annotator error. We introduce an ambiguity measure that maps a discrete response distribution to a scalar in the unit interval, designed to quantify aleatoric uncertainty in categorical tasks. The measure bears a close relationship to quadratic entropy (Gini-style impurity) but departs from those indices by treating an explicit "can't solve" category asymmetrically, thereby separating uncertainty arising from class-level indistinguishability from uncertainty due to explicit unresolvability. We analyze the measure's formal properties and contrast its behavior with a representative ambiguity measure from the literature. Moving beyond description, we develop statistical tools for inference: we propose frequentist point estimators for population ambiguity and derive the Bayesian posterior over ambiguity induced by Dirichlet priors on the underlying probability vector, providing a principled account of epistemic uncertainty. Numerical examples illustrate estimation, calibration, and practical use for dataset-quality assessment and downstream machine-learning workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。