研究数据收集与标注方式如何影响网络欺凌检测模型的泛化能力。
Exploration and Evaluation of Bias in Cyberbullying Detection with Machine Learning
- 分析三种常见数据集的偏差来源，包括定义差异和标注方法。
- 跨数据集测试显示模型宏观F1分数平均下降0.222。
- 强调数据清洗与跨数据集验证对实际应用的重要性。
机器学习模型的有效性依赖于其对未见数据的泛化能力。本研究基于三个流行的网络欺凌检测数据集，探究数据本身、采集方式及标注过程对模型性能的影响。重点讨论了不同网络欺凌定义和数据采集方式带来的偏差，并深入分析了数据集扩展方法（利用现有数据点获取并标注新样本）的影响。通过跨数据集评估，明确测试模型在未见数据上的泛化能力。结果表明，模型宏观F1分数平均下降0.222，证实了数据集构建质量与跨数据集验证对模型真实可用性的关键作用。实验代码及相关资源详见https://github.com/rootdrew27/cyberbullying-ml。
原文摘要 · Abstract (English)
It is well known that the usefulness of a machine learning model is due to its ability to generalize to unseen data. This study uses three popular cyberbullying datasets to explore the effects of data, how it's collected, and how it's labeled, on the resulting machine learning models. The bias introduced from differing definitions of cyberbullying and from data collection is discussed in detail. An emphasis is made on the impact of dataset expansion methods, which utilize current data points to fetch and label new ones. Furthermore, explicit testing is performed to evaluate the ability of a model to generalize to unseen datasets through cross-dataset evaluation. As hypothesized, the models have a significant drop in the Macro F1 Score, with an average drop of 0.222. As such, this study effectively highlights the importance of dataset curation and cross-dataset testing for creating models with real-world applicability. The experiments and other code can be found at https://github.com/rootdrew27/cyberbullying-ml.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。