用原型学习提升标注者建模,更准更省资源
Understanding the Essence: Delving into Annotator Prototype Learning for Multi-Class Annotation Aggregation
- 用原型矩阵集合代替单一混淆矩阵,捕捉标注者多元能力
- 在11个真实数据集上平均提升3%准确率,最高达15%
- 特别适合标注稀疏或类别不均衡场景,计算成本降90%以上
多类别标注显著推动了人工智能应用发展,真值推断是聚合噪声和偏差标注的关键技术。现有最先进方法通常用混淆矩阵建模每个标注者的专长,但存在两大问题:当多数标注者仅标注少数任务,或类别不平衡时,估计的混淆矩阵不可靠;且单个混淆矩阵难以完整刻画标注者在所有任务上的专长模式。为此,本文提出一种基于原型学习的混淆矩阵方法PTBCC(ProtoType learning-driven Bayesian Classifier Combination),通过引入一组原型混淆矩阵来捕捉所有标注者的内在专长模式。不再使用单一混淆矩阵,而是将每位标注者的专长表示为这些原型上的狄利克雷先验分布。该机制有效缓解数据稀疏与类别不平衡问题,实现更丰富灵活的标注者表征。在11个真实世界数据集上的大量实验表明,PTBCC在最佳情况下可提升15%准确率,平均准确率提高3%,同时计算成本降低超过90%。
原文摘要 · Abstract (English)
Multi-class classification annotations have significantly advanced AI applications, with truth inference serving as a critical technique for aggregating noisy and biased annotations. Existing state-of-the-art methods typically model each annotator's expertise using a confusion matrix. However, these methods suffer from two widely recognized issues: 1) when most annotators label only a few tasks, or when classes are imbalanced, the estimated confusion matrices are unreliable, and 2) a single confusion matrix often remains inadequate for capturing each annotator's full expertise patterns across all tasks. To address these issues, we propose a novel confusion-matrix-based method, PTBCC (ProtoType learning-driven Bayesian Classifier Combination), to introduce a reliable and richer annotator estimation by prototype learning. Specifically, we assume that there exists a set $S$ of prototype confusion matrices, which capture the inherent expertise patterns of all annotators. Rather than a single confusion matrix, the expertise per annotator is extended as a Dirichlet prior distribution over these prototypes. This prototype learning-driven mechanism circumvents the data sparsity and class imbalance issues, ensuring a richer and more flexible characterization of annotators. Extensive experiments on 11 real-world datasets demonstrate that PTBCC achieves up to a 15% accuracy improvement in the best case, and a 3% higher average accuracy while reducing computational cost by over 90%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。