构建首个孟加拉语毒言净化平行语料库，提升可解释性净化效果
BanglaNirTox: A Large-scale Parallel Corpus for Explainable AI in Bengali Text Detoxification
- 用帕累托优化LLM+思维链提示生成净化文本
- 创建68,041条带毒性标签与理由的孟加拉语语料
- 适合研究低资源语言净化与可解释AI的学者
孟加拉语中的有毒语言在在线环境中仍普遍存在，但相关防护措施有限。尽管高资源语言的文本净化已取得进展，孟加拉语因资源匮乏而研究不足。本文提出一种结合帕累托优化大语言模型（LLMs）与思维链（CoT）提示的新型净化管道，生成净化后的句子。为支持该工作，我们构建了BanglaNirTox，一个包含68,041条人工生成的孟加拉语有毒句及其类别级毒性标签、推理过程和净化改写版本的平行语料库，使用帕累托优化的LLM在随机样本上评估生成质量。该语料库用于微调语言模型，以生成更优的孟加拉语净化文本。实验表明，帕累托优化的LLM配合CoT提示能显著提升孟加拉语文本净化的质量与一致性。
原文摘要 · Abstract (English)
Toxic language in Bengali remains prevalent, especially in online environments, with few effective precautions against it. Although text detoxification has seen progress in high-resource languages, Bengali remains underexplored due to limited resources. In this paper, we propose a novel pipeline for Bengali text detoxification that combines Pareto class-optimized large language models (LLMs) and Chain-of-Thought (CoT) prompting to generate detoxified sentences. To support this effort, we construct BanglaNirTox, an artificially generated parallel corpus of 68,041 toxic Bengali sentences with class-wise toxicity labels, reasonings, and detoxified paraphrases, using Pareto-optimized LLMs evaluated on random samples. The resulting BanglaNirTox dataset is used to fine-tune language models to produce better detoxified versions of Bengali sentences. Our findings show that Pareto-optimized LLMs with CoT prompting significantly enhance the quality and consistency of Bengali text detoxification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。