arXiv:2509.06838cs.CLcs.CR2025-09

评测大模型在波斯语文化下的可信度,发现安全问题最突出。

EPT Benchmark: Evaluation of Persian Trustworthiness in Large Language Models

  • 构建波斯文化适配的可信度评估基准,涵盖六维度
  • 多模型测试显示安全性能显著不足
  • 适合关注本地化AI伦理与文化对齐的研究者

大型语言模型在多种语言任务中表现卓越,但其可信度仍是关键挑战。本文提出针对波斯语文化的EPT(可信度评估)基准,从真实性、安全性、公平性、鲁棒性、隐私和伦理对齐六个维度评估主流模型。研究收集了标注数据集,对ChatGPT、Claude、DeepSeek、Gemini、Grok、LLaMA、Mistral和Qwen等模型进行了自动化与人工评估。结果揭示安全维度存在显著缺陷,反映模型行为与波斯伦理文化价值观存在偏差。研究为构建更可信、文化敏感的AI系统提供重要依据。数据集已公开:https://github.com/Rezamirbagheri110/EPT-Benchmark。

原文摘要 · Abstract (English)

Large Language Models (LLMs), trained on extensive datasets using advanced deep learning architectures, have demonstrated remarkable performance across a wide range of language tasks, becoming a cornerstone of modern AI technologies. However, ensuring their trustworthiness remains a critical challenge, as reliability is essential not only for accurate performance but also for upholding ethical, cultural, and social values. Careful alignment of training data and culturally grounded evaluation criteria are vital for developing responsible AI systems. In this study, we introduce the EPT (Evaluation of Persian Trustworthiness) metric, a culturally informed benchmark specifically designed to assess the trustworthiness of LLMs across six key aspects: truthfulness, safety, fairness, robustness, privacy, and ethical alignment. We curated a labeled dataset and evaluated the performance of several leading models - including ChatGPT, Claude, DeepSeek, Gemini, Grok, LLaMA, Mistral, and Qwen - using both automated LLM-based and human assessments. Our results reveal significant deficiencies in the safety dimension, underscoring the urgent need for focused attention on this critical aspect of model behavior. Furthermore, our findings offer valuable insights into the alignment of these models with Persian ethical-cultural values and highlight critical gaps and opportunities for advancing trustworthy and culturally responsible AI. The dataset is publicly available at: https://github.com/Rezamirbagheri110/EPT-Benchmark.

大模型评估文化对齐可信度波斯语

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。