用少量上下文提升低资源巴尔干语毒液内容检测效果
Large Language Models for Toxic Language Detection in Low-Resource Balkan Languages
- 在零样本和加上下文两种模式下测试大模型
- 加上下文使召回率平均提升0.12，F1最高达0.82
- 适合关注小语种毒性内容检测的研究者与平台方
在线毒液语言造成真实伤害，尤其在缺乏内容审核工具的地区。本研究评估大型语言模型在塞尔维亚语、克罗地亚语和波斯尼亚语（数据标注有限）中的毒液评论识别能力。我们构建并人工标注了4,500条来自音乐、政治、体育、模特、网红内容、性别议题及一般话题的YouTube和TikTok评论数据集。测试了GPT-3.5 Turbo、GPT-4.1、Gemini 1.5 Pro和Claude 3 Opus四种模型，采用零样本和上下文增强两种模式，衡量精确率、召回率、F1分数、准确率和误报率。加入简短上下文片段使平均召回率提升约0.12，F1最高提升0.10，但有时增加误报。最佳平衡由上下文增强模式下的Gemini达成，F1为0.82，准确率为0.82；零样本下GPT-4.1在精确率上最优，误报最低。结果表明，仅通过优化提示设计即可显著提升低资源语境下的毒液检测性能。
原文摘要 · Abstract (English)
Online toxic language causes real harm, especially in regions with limited moderation tools. In this study, we evaluate how large language models handle toxic comments in Serbian, Croatian, and Bosnian, languages with limited labeled data. We built and manually labeled a dataset of 4,500 YouTube and TikTok comments drawn from videos across diverse categories, including music, politics, sports, modeling, influencer content, discussions of sexism, and general topics. Four models (GPT-3.5 Turbo, GPT-4.1, Gemini 1.5 Pro, and Claude 3 Opus) were tested in two modes: zero-shot and context-augmented. We measured precision, recall, F1 score, accuracy and false positive rates. Including a short context snippet raised recall by about 0.12 on average and improved F1 score by up to 0.10, though it sometimes increased false positives. The best balance came from Gemini in context-augmented mode, reaching an F1 score of 0.82 and accuracy of 0.82, while zero-shot GPT-4.1 led on precision and had the lowest false alarms. We show how adding minimal context can improve toxic language detection in low-resource settings and suggest practical strategies such as improved prompt design and threshold calibration. These results show that prompt design alone can yield meaningful gains in toxicity detection for underserved Balkan language communities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。