中文安全问答基准，评估大模型在法律政策等领域的事实准确性。
Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models
- 专为中文大模型设计的安全性短问题评测集
- 覆盖多领域，测试模型对敏感话题的事实回答能力
- 适合关注AI合规与安全部署的研究者使用
随着大语言模型的快速发展，安全性问题日益突出。模型的安全性与其对法律、政策、伦理等领域安全知识的理解准确度、全面性和清晰度密切相关。这种事实性能力直接影响模型在特定区域部署和应用的安全性与合规性。为解决这一问题，我们提出中文SafetyQA基准，具备中文、多样化、高质量、静态、易评估、安全相关、无害等特性。基于该基准，我们对现有大模型的事实性能力进行了全面评估，并分析其与检索增强生成（RAG）能力及抗攻击鲁棒性的关系。
原文摘要 · Abstract (English)
With the rapid advancement of Large Language Models (LLMs), significant safety concerns have emerged. Fundamentally, the safety of large language models is closely linked to the accuracy, comprehensiveness, and clarity of their understanding of safety knowledge, particularly in domains such as law, policy and ethics. This factuality ability is crucial in determining whether these models can be deployed and applied safely and compliantly within specific regions. To address these challenges and better evaluate the factuality ability of LLMs to answer short questions, we introduce the Chinese SafetyQA benchmark. Chinese SafetyQA has several properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate, Safety-related, Harmless). Based on Chinese SafetyQA, we perform a comprehensive evaluation on the factuality abilities of existing LLMs and analyze how these capabilities relate to LLM abilities, e.g., RAG ability and robustness against attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。