首个针对乌尔都语毒害片段检测的精准标注框架，提升社交媒体内容安全
MUTEX: Leveraging Multilingual Transformers and Conditional Random Fields for Enhanced Urdu Toxic Span Detection
- 结合多语言Transformer与条件随机场进行词级毒性片段识别
- 在社交媒体等多领域数据上达60%词级F1，首次建立乌尔都语毒害片段检测基线
- 有效应对乌尔都语复杂语法、混用语言和形态变化等挑战，适合内容审核场景
乌尔都语毒害片段检测受限于现有系统多依赖句子级分类，难以定位具体毒害片段。这一问题因缺乏词级标注资源、乌尔都语语言复杂性、频繁的语言混用、非正式表达及丰富的形态变化而加剧。本文提出MUTEX：一种结合多语言Transformer与条件随机场（CRF）的乌尔都语毒害片段检测框架，利用人工标注的词级毒害片段数据集提升性能与可解释性。MUTEX采用XLM RoBERTa并加入CRF层进行序列标注，在来自社交媒体、在线新闻和YouTube评论的多领域数据上测试，以词级F1为评估指标。结果表明，MUTEX达到60%的词级F1，是首个乌尔都语毒害片段检测的监督基线。进一步分析显示，基于Transformer的模型能更有效隐式捕捉上下文毒性，且在应对语言混用与形态变异方面优于其他模型。
原文摘要 · Abstract (English)
Urdu toxic span detection remains limited because most existing systems rely on sentence-level classification and fail to identify the specific toxic spans within those text. It is further exacerbated by the multiple factors i.e. lack of token-level annotated resources, linguistic complexity of Urdu, frequent code-switching, informal expressions, and rich morphological variations. In this research, we propose MUTEX: a multilingual transformer combined with conditional random fields (CRF) for Urdu toxic span detection framework that uses manually annotated token-level toxic span dataset to improve performance and interpretability. MUTEX uses XLM RoBERTa with CRF layer to perform sequence labeling and is tested on multi-domain data extracted from social media, online news, and YouTube reviews using token-level F1 to evaluate fine-grained span detection. The results indicate that MUTEX achieves 60% token-level F1 score that is the first supervised baseline for Urdu toxic span detection. Further examination reveals that transformer-based models are more effective at implicitly capturing the contextual toxicity and are able to address the issues of code-switching and morphological variation than other models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。