为政府AI系统设计统一评估框架，提升生成式AI的可信赖性
Generative Responsible AI Data Evaluation Schema (GRAIDES) for AI Assurance in Local Government
- 提出轻量级数据模型GRAIDES，整合多厂商AI评估数据
- 在西敏市议会案例中实现评估者间系统性分歧检测
- 提供代码与架构蓝图，助力组织级AI安全验证
生成式人工智能的应用信任依赖于可度量、可治理的性能与安全证据。然而实践中，评估数据常分散于各系统，结构不一且难以比较。本文提出生成式负责任AI数据评估框架（GRAIDES），一种轻量级开源数据模型，用于整合主流厂商的AI可观测性数据。通过共享代码、架构及统计评估的实践指南，指导组织层面的生成式系统保障。基于西敏市议会的案例研究，重点测量人-模型对齐情况，识别评估者间的系统性分歧。将评估视为数据建模问题，GRAIDES为生成式AI系统的基准测试、调优与保障活动提供了更一致、可复现的实践路径。
原文摘要 · Abstract (English)
Trust in the application of generative Artificial Intelligence (AI) relies on well-governed measurable evidence of performance and safety. In practice, however, evaluation data is often fragmented across systems, inconsistently structured and difficult to compare. We introduce the Generative Responsible AI Data Evaluation Schema (GRAIDES) as a lightweight open-source data model for centralising AI observability across popular vendors. Practical blueprints for code, architecture and statistical evaluation are shared as guidance about how to approach generative system assurance at the organisational level. Illustrative case study results are reported from Westminster City Council's AI catalogue with a focus on measuring human-model alignment including detecting systematic disagreement between evaluators. By framing evaluations as a data modelling problem, GRAIDES provides a practical pathway toward more consistent and reproducible benchmarking, tuning and assurance activities for generative AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。