厘清AI伦理评估与系统风险的关联，推动全面评价体系构建
Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms
- 从系统属性、风险和伤害出发，重构伦理评估框架
- 800项评估指标中仅4类原则受关注，多聚焦模型输出层
- 揭示当前评估碎片化问题，适合政策制定与工业界参考
过去十年间，围绕AI系统社会与伦理影响的评估体系不断涌现，主要依据高层级伦理原则构建。然而这些评估方法分散使用，缺乏对实际系统结构的关注。本文基于涵盖近800项指标的系统性文献综述，分析其与11项AI伦理原则的对应关系。发现多数指标集中于公平性、透明性、隐私与信任四类原则，且主要针对模型或输出组件。极少考虑系统各要素间的交互作用，每种伤害类型所覆盖的风险范围也极为有限。许多评估指标与伤害发生位置脱节，且缺乏设定合理阈值的指导。这表明当前评估仍处于碎片化状态，仅孤立测量局部环节，未能捕捉伤害在系统中的动态演化过程。将评估措施与系统属性、潜在风险及实际伤害相联结，有助于加强监管、推动产业落地，并为未来研究提供系统性基础。
原文摘要 · Abstract (English)
Over the past decade, an ecosystem of measures has emerged to evaluate the social and ethical implications of AI systems, largely shaped by high-level ethics principles. These measures are developed and used in fragmented ways, without adequate attention to how they are situated in AI systems. In this paper, we examine how existing measures used in the computing literature map to AI system components, attributes, hazards, and harms. Our analysis draws on a scoping review resulting in nearly 800 measures corresponding to 11 AI ethics principles. We find that most measures focus on four principles - fairness, transparency, privacy, and trust - and primarily assess model or output system components. Few measures account for interactions across system elements, and only a narrow set of hazards is typically considered for each harm type. Many measures are disconnected from where harm is experienced and lack guidance for setting meaningful thresholds. These patterns reveal how current evaluation practices remain fragmented, measuring in pieces rather than capturing how harms emerge across systems. Framing measures with respect to system attributes, hazards, and harms can strengthen regulatory oversight, support actionable practices in industry, and ground future research in systems-level understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。