混合语言嵌入能更好识别多语种仇恨言论。
On Importance of Code-Mixed Embeddings for Hate Speech Identification
- 用印地-英语混合语料训练嵌入模型，提升多语种理解能力。
- HingBERT在仇恨言论数据集上表现优于BERT和普通FastText。
- 适合研究多语种社会中的语言多样性与内容安全问题。
代码混用指在单句中使用两种或以上语言，常见于印度等多语言社群。传统NLP工具基于单一语言数据训练，在处理混合语言文本时面临挑战，尤其在仇恨言论检测任务中，受语言差异、文化背景和数据稀疏性影响，信息提取困难。为此，我们分析了代码混用嵌入的重要性，并评估了在印地-英语语料库L3Cube-HingCorpus上训练的BERT与HingBERT模型在仇恨言论检测中的表现。实验表明，得益于大规模混合语料训练，HingBERT模型在测试集上优于BERT；同时，基于混合语料训练的Hing-FastText也优于标准英文FastText与原始BERT模型。
原文摘要 · Abstract (English)
Code-mixing is the practice of using two or more languages in a single sentence, which often occurs in multilingual communities such as India where people commonly speak multiple languages. Classic NLP tools, trained on monolingual data, face challenges when dealing with code-mixed data. Extracting meaningful information from sentences containing multiple languages becomes difficult, particularly in tasks like hate speech detection, due to linguistic variation, cultural nuances, and data sparsity. To address this, we aim to analyze the significance of code-mixed embeddings and evaluate the performance of BERT and HingBERT models (trained on a Hindi-English corpus) in hate speech detection. Our study demonstrates that HingBERT models, benefiting from training on the extensive Hindi-English dataset L3Cube-HingCorpus, outperform BERT models when tested on hate speech text datasets. We also found that code-mixed Hing-FastText performs better than standard English FastText and vanilla BERT models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。