arXiv:2608.22894cs.CLcs.AI2026-08

构建多方言阿拉伯语去有害化数据集,支持安全文本生成研究

AraDetox: A Multi-Dialect Arabic Detoxification Dataset

  • 用大模型生成10.5万条有害内容改写版本,覆盖四种阿拉伯语方言
  • 改写后语义保留率高,有害内容清除率达98%以上,风格匹配原方言
  • 适合从事阿拉伯语安全生成、多语言NLP及有害内容检测的研究者

阿拉伯语有害语言检测受到广泛关注,但去有害化研究仍不充分。本文提出AraDetox,一个包含10,500条有害社交媒体帖子及其84,000条通过GPT-5和Gemini 2.5 Flash生成的改写版本的多方言去有害化数据集,涵盖现代标准阿拉伯语、海湾、黎凡特和埃及方言。生成结果经人工评估与自动分析(词汇变化、语义保留、情感倾向、方言风格)验证:去有害化主要为语义保持重写任务,词汇与结构大幅重构,但语义相似度始终较高。人工评估确认有害内容有效移除,同时基本保留原文意义。方言分析显示生成样本与参考方言语料在风格上具可测量一致性。对比现有资源揭示两类互补方法:最小编辑替换与语义保持重构。研究证明可通过大模型辅助生成结合人工验证构建大规模阿拉伯语去有害化资源。数据集已公开于https://github.com/ArabicNLP-UK/AraDetox,以支持未来阿拉伯语去有害化、安全文本生成及多方言NLP研究。

原文摘要 · Abstract (English)

Arabic harmful-language detection has received considerable attention, yet Arabic text detoxification remains underexplored. We introduce AraDetox, a multi-dialect Arabic detoxification dataset comprising 10,500 harmful social-media posts and 84,000 detoxified rewrites generated using GPT-5 and Gemini 2.5 Flash across Modern Standard Arabic, Gulf, Levantine, and Egyptian Arabic. The generated outputs were assessed through human evaluation and automatic analyses of lexical change, semantic preservation, sentiment, and dialectal style. Results show that detoxification is primarily a meaning-preserving rewriting task: substantial lexical and structural reformulation is accompanied by consistently high semantic similarity. Human evaluation confirms successful harmful-language removal while largely preserving the original meaning. Dialectal analyses further indicate that the generated variants exhibit measurable stylistic alignment with reference Arabic dialect corpora. Comparison with existing resources highlights two complementary approaches to detoxification: minimal-edit lexical substitution and meaning-preserving reformulation. Our findings demonstrate that large-scale Arabic detoxification resources can be constructed through LLM-assisted generation and human verification. The dataset is publicly available at https://github.com/ArabicNLP-UK/AraDetox to support future research on Arabic detoxification, safe text generation, and multi-dialect Arabic NLP.

去有害化多方言文本生成阿拉伯语

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。