用真实对话数据评估大模型的伦理表现。
RAIL in the Wild: Operationalizing Responsible AI Evaluation Using Anthropic's Value Dataset
- 构建八维度可量化评估体系，分析大模型伦理行为。
- 基于30.8万条对话和3000+价值标注，生成合成评分。
- 为实际应用中的责任AI提供可操作评估方案，适合政策与工程团队参考。
随着大语言模型在现实应用中日益普及，确保其符合伦理标准至关重要。现有伦理框架虽强调公平、透明与问责，但缺乏可操作的评估方法。本文引入负责任AI实验室（RAIL）框架，包含八个可量化的维度，用于评估大模型的规范行为。研究基于Anthropic的「价值在野外」数据集，该数据集包含超过308,000条匿名对话及3,000多个标注的价值表达。通过将这些价值映射至RAIL维度，计算合成评分，揭示了大模型在真实使用中的伦理表现，为责任AI的实际落地提供可操作的评估路径。
原文摘要 · Abstract (English)
As AI systems become embedded in real-world applications, ensuring they meet ethical standards is crucial. While existing AI ethics frameworks emphasize fairness, transparency, and accountability, they often lack actionable evaluation methods. This paper introduces a systematic approach using the Responsible AI Labs (RAIL) framework, which includes eight measurable dimensions to assess the normative behavior of large language models (LLMs). We apply this framework to Anthropic's "Values in the Wild" dataset, containing over 308,000 anonymized conversations with Claude and more than 3,000 annotated value expressions. Our study maps these values to RAIL dimensions, computes synthetic scores, and provides insights into the ethical behavior of LLMs in real-world use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。