首个针对非洲语言的越狱测试基准，揭示本地化对模型安全的影响
TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages
- 构建七种非洲语言的越狱测试集，融合文化语境与多语言混用
- 非洲语言提问时拒绝率下降37%，文化适配提示最易突破安全防线
- 发现低资源语言模型理解力差、人工评估可靠性低，适合安全研究者使用
大型语言模型（LLMs）的安全性评估仍以英语为主，导致低资源语言（尤其是非洲语言）严重缺乏研究。本文提出TUKABENCH，一个面向七种非洲语言的越狱基准，通过四种设置扩展JailbreakBench（JBB）：人类翻译的JBB提示、将英语提示适配非洲语境后人工翻译、经GPT-5.2交互验证的人工精选提示，以及中英文混用的提示，分别隔离语言、文化背景和提示隐蔽性对模型安全的影响。在封闭与开放模型上测试发现，使用非洲语言提问时拒绝率比英语降低37%，文化适配提示的拒绝率最低。评估还揭示两个结构性问题：模型理解能力不足，以及低资源语言下LLM作为评判者的可靠性下降。为此引入「偏移」（Deflection）指标，并通过人工标注验证，发现判别结果与人工一致性在低资源语言和非主流书写系统中显著降低。
原文摘要 · Abstract (English)
Safety evaluation of Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African ones, critically underexplored. We introduce TUKABENCH, a jailbreak benchmark for seven African languages that extends JailbreakBench (JBB) beyond direct translation through four settings: human translation of JBB prompts, English adaptation to African contexts followed by human translation, human-curated prompts validated through interactions with GPT-5.2, and code-switched prompts combining English and African languages, isolating the effect of language, cultural grounding, and prompt evasiveness on model safety. Across closed and open models, prompting in African languages reduces refusal relative to English, with culturally adapted prompts leading to least refusal. The evaluation also surfaces two structural limitations: model comprehension failures and reduced LLM-as-a-judge reliability in LRLs. To capture the first, we introduce Deflection alongside Refused and Jailbroken; to assess the second, we validate outputs with human annotations, showing that judge-human agreement drops in lower-resource languages and less commonly supported scripts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。