测试大模型对不同方言的毒性判断一致性，发现人机判断差异最大
Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties
- 构建涵盖60种方言的多语种毒性数据集，通过合成与人工翻译生成
- 大模型在跨语言和方言判断中表现敏感，但人机一致性最弱
- 适合关注AI评估公平性、跨语言内容安全的研究者
目前关于方言差异如何影响现代大模型毒性检测的研究仍不系统。尽管使用大模型作为评估者（LLM-as-a-judge）是新兴方向，但其对方言细微差别的敏感性尚未充分探索。本文通过全面评估大模型在多种方言上的毒性判断能力，填补这一空白。我们基于合成转换和人工辅助翻译，构建了一个覆盖10个语言集群和60种方言的多方言数据集。随后评估了三种大模型在多语言、方言及大模型-人类一致性方面的表现。结果表明，大模型对多语言和方言变化具有敏感性；但在一致性排序中，大模型与人类判断的一致性最弱，其次为方言内部一致性。代码仓库： https://github.com/ffaisal93/dialect_toxicity_llm_judge
原文摘要 · Abstract (English)
There has been little systematic study on how dialectal differences affect toxicity detection by modern LLMs. Furthermore, although using LLMs as evaluators ("LLM-as-a-judge") is a growing research area, their sensitivity to dialectal nuances is still underexplored and requires more focused attention. In this paper, we address these gaps through a comprehensive toxicity evaluation of LLMs across diverse dialects. We create a multi-dialect dataset through synthetic transformations and human-assisted translations, covering 10 language clusters and 60 varieties. We then evaluated three LLMs on their ability to assess toxicity across multilingual, dialectal, and LLM-human consistency. Our findings show that LLMs are sensitive in handling both multilingual and dialectal variations. However, if we have to rank the consistency, the weakest area is LLM-human agreement, followed by dialectal consistency. Code repository: \url{https://github.com/ffaisal93/dialect_toxicity_llm_judge}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。