arXiv:2507.23756cs.LGcs.HC2025-07

考虑情绪与疲劳的推荐系统提升标注效率,减少错误。

Improving annotator selection in Active Learning using a mood and fatigue-aware Recommender System

  • 用知识库推荐系统综合过往准确率、情绪和疲劳度选标注员。
  • 相比不考虑心理状态,错误率降低,模型不确定性减少。
  • 适合关注人因因素的主动学习研究者或平台设计者。

本研究针对主动学习(AL)中选择最优标注员的挑战,旨在减少误分类。尽管AL能降低标注成本与时间,但仍需提升标注精度以更快达到目标性能。现有策略多忽略影响标注效率的内在因素,如情绪、注意力、动机和疲劳水平。本文通过引入基于知识的推荐系统(RS),结合标注员的历史准确率、当前情绪与疲劳状态,以及待标注样本特征,动态推荐最佳标注员。基于情绪与疲劳对人类表现影响的文献,模拟真实标注场景并预测其表现。实验表明,融合内部因素后,标注错误显著减少,模型训练过程中的不确定性下降,且准确率与F1分数均有提升,虽增幅有限但意义明确。该研究为探索认知因素影响主动学习提供了新思路。

原文摘要 · Abstract (English)

This study centers on overcoming the challenge of selecting the best annotators for each query in Active Learning (AL), with the objective of minimizing misclassifications. AL recognizes the challenges related to cost and time when acquiring labeled data, and decreases the number of labeled data needed. Nevertheless, there is still the necessity to reduce annotation errors, aiming to be as efficient as possible, to achieve the expected accuracy faster. Most strategies for query-annotator pairs do not consider internal factors that affect productivity, such as mood, attention, motivation, and fatigue levels. This work addresses this gap in the existing literature, by not only considering how the internal factors influence annotators (mood and fatigue levels) but also presenting a new query-annotator pair strategy, using a Knowledge-Based Recommendation System (RS). The RS ranks the available annotators, allowing to choose one or more to label the queried instance using their past accuracy values, and their mood and fatigue levels, as well as information about the instance queried. This work bases itself on existing literature on mood and fatigue influence on human performance, simulating annotators in a realistic manner, and predicting their performance with the RS. The results show that considering past accuracy values, as well as mood and fatigue levels reduces the number of annotation errors made by the annotators, and the uncertainty of the model through its training, when compared to not using internal factors. Accuracy and F1-score values were also better in the proposed approach, despite not being as substantial as the aforementioned. The methodologies and findings presented in this study begin to explore the open challenge of human cognitive factors affecting AL.

主动学习人因因素推荐系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。