构建首个泰语文化安全评测基准，揭示中文模型在本土攻击下的脆弱性
ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts
- 设计1954个融合泰国文化语境的恶意提示，覆盖本地化攻击场景
- 发现闭源模型安全性能显著优于开源模型，泰语特化攻击成功率更高
- 开源分类器达84.4%准确率，支持社区复现与持续评测
大型语言模型的安全评估长期聚焦英文，忽视非英语语言及文化背景风险。本文针对泰语与泰国文化，提出首个开源基准ThaiSafetyBench，包含1954个泰语恶意提示，涵盖通用有害内容及具泰国文化、社会和语境特性的攻击。使用该基准评估24个模型，以GPT-4.1和Gemini-2.5-Pro作为评判者。结果表明，闭源模型整体安全表现优于开源模型，且泰语特化攻击的攻击成功率（ASR）显著高于通用泰语攻击，暴露当前对齐方法在文化敏感场景下的不足。为提升可复现性与效率，我们微调基于DeBERTa的有害响应分类器ThaiSafetyClassifier，其加权F1得分为84.4%，媲美GPT-4.1判断。项目公开训练权重与脚本，并建立持续更新的ThaiSafetyBench排行榜，推动社区参与。
原文摘要 · Abstract (English)
The safety evaluation of large language models (LLMs) remains largely centered on English, leaving non-English languages and culturally grounded risks underexplored. In this work, we investigate LLM safety in the context of the Thai language and culture and introduce ThaiSafetyBench, an open-source benchmark comprising 1,954 malicious prompts written in Thai. The dataset covers both general harmful prompts and attacks that are explicitly grounded in Thai cultural, social, and contextual nuances. Using ThaiSafetyBench, we evaluate 24 LLMs, with GPT-4.1 and Gemini-2.5-Pro serving as LLM-as-a-judge evaluators. Our results show that closed-source models generally demonstrate stronger safety performance than open-source counterparts, raising important concerns regarding the robustness of openly available models. Moreover, we observe a consistently higher Attack Success Rate (ASR) for Thai-specific, culturally contextualized attacks compared to general Thai-language attacks, highlighting a critical vulnerability in current safety alignment methods. To improve reproducibility and cost efficiency, we further fine-tune a DeBERTa-based harmful response classifier, which we name ThaiSafetyClassifier. The model achieves a weighted F1 score of 84.4%, matching GPT-4.1 judgments. We publicly release the fine-tuning weights and training scripts to support reproducibility. Finally, we introduce the ThaiSafetyBench leaderboard to provide continuously updated safety evaluations and encourage community participation. - ThaiSafetyBench HuggingFace Dataset: https://huggingface.co/datasets/typhoon-ai/ThaiSafetyBench - ThaiSafetyBench Github: https://github.com/trapoom555/ThaiSafetyBench - ThaiSafetyClassifier HuggingFace Model: https://huggingface.co/typhoon-ai/ThaiSafetyClassifier - ThaiSafetyBench Leaderboard: https://huggingface.co/spaces/typhoon-ai/ThaiSafetyBench-Leaderboard
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。