为两种资源匮乏的非洲语言开发了可解释的文本净化工具。
Text Detoxification in isiXhosa and Yorùbá: A Cross-Lingual Machine Learning Approach for Low-Resource African Languages
- 用TF-IDF与逻辑回归实现透明化毒性检测，结合词典和标记引导重写。
- 检测准确率61-86%，重写后所有有毒句被净化且无误伤。
- 适合关注非洲语言安全工具、低资源场景的开发者与研究者。
毒性语言是在线安全参与的主要障碍，但针对非洲语言的可靠缓解工具仍极为稀缺。本研究填补这一空白，针对两种低资源非洲语言 isiXhosa 与 Yorùbá，探索自动文本净化（将有毒文本转为中性）技术。提出一种新颖实用的混合方法：采用轻量级、可解释的 TF-IDF 与逻辑回归模型进行透明毒性检测，并设计受控的词典与标记引导重写模块。构建了一个平行语料库，涵盖习语、变音符号及代码切换现象，用于模型训练与评估。检测组件在 isiXhosa 上达到 61-72% 的分层交叉验证准确率，Yorùbá 达 72-86%，单语言 ROC-AUC 最高达 0.88。重写模块成功净化所有检测出的有毒句子，同时 100% 保留非有毒内容。结果表明，可扩展、可解释的机器学习检测结合规则化编辑，为文化适配的安全工具提供了高效可行方案，确立了非洲语言低资源文本风格迁移的新基准。
原文摘要 · Abstract (English)
Toxic language is one of the major barrier to safe online participation, yet robust mitigation tools are scarce for African languages. This study addresses this critical gap by investigating automatic text detoxification (toxic to neutral rewriting) for two low-resource African languages, isiXhosa and Yorùbá. The work contributes a novel, pragmatic hybrid methodology: a lightweight, interpretable TF-IDF and Logistic Regression model for transparent toxicity detection, and a controlled lexicon- and token-guided rewriting component. A parallel corpus of toxic to neutral rewrites, which captures idiomatic usage, diacritics, and code switching, was developed to train and evaluate the model. The detection component achieved stratified K-fold accuracies of 61-72% (isiXhosa) and 72-86% (Yorùbá), with per-language ROC-AUCs up to 0.88. The rewriting component successfully detoxified all detected toxic sentences while preserving 100% of non-toxic sentences. These results demonstrate that scalable, interpretable machine learning detectors combined with rule-based edits offer a competitive and resource-efficient solution for culturally adaptive safety tooling, setting a new benchmark for low-resource Text Style Transfer (TST) in African languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。