首个统一评估框架，可检验多标注者倾向学习是否真实捕捉个体标注行为。
A Unified Evaluation Framework for Multi-Annotator Tendency Learning
- 提出双指标评估框架，量化模型对标注者倾向的捕捉能力
- 在多个数据集上验证框架有效性，证明其能准确反映标注者行为差异
- 适合研究标注者偏见、可解释性分析的学者使用
近期多标注者学习研究从以共识为导向（CoL）转向个体倾向学习（ITL），后者建模标注者的个性化标注行为模式（即倾向），以解释标注决策。然而，当前尚无有效评估框架来判断ITL方法是否真正捕获了个体倾向并提供有意义的行为解释。为此，我们提出首个统一评估框架，包含两个新指标：(1) 标注者间一致性差异（DIC），通过比较模型预测的标注者间相似性结构与真实结构，衡量倾向捕捉程度；(2) 行为对齐可解释性（BAE），利用多维缩放（MDS）将可解释性推导出的相似性与真实标注相似性对齐，评估解释与行为的相关性。大量实验验证了该框架的有效性。
原文摘要 · Abstract (English)
Recent works have emerged in multi-annotator learning that shift focus from Consensus-oriented Learning (CoL), which aggregates multiple annotations into a single ground-truth prediction, to Individual Tendency Learning (ITL), which models annotator-specific labeling behavior patterns (i.e., tendency) to provide explanation analysis for understanding annotator decisions. However, no evaluation framework currently exists to assess whether ITL methods truly capture individual tendencies and provide meaningful behavioral explanations. To address this gap, we propose the first unified evaluation framework with two novel metrics: (1) Difference of Inter-annotator Consistency (DIC) quantifies how well models capture annotator tendencies by comparing predicted inter-annotator similarity structures with ground-truth; (2) Behavior Alignment Explainability (BAE) evaluates how well model explanations reflect annotator behavior and decision relevance by aligning explainability-derived with ground-truth labeling similarity structures via Multidimensional Scaling (MDS). Extensive experiments validate the effectiveness of our proposed evaluation framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。