用五维度评分卡评估AI数据质量，提升透明与可信度。
AI Data Development: A Scorecard for the System Card Framework
- 基于系统卡片框架，从数据字典等五方面结构化评分。
- 在四个真实数据集上验证，发现可改进点并生成定制建议。
- 兼顾技术与伦理，适合数据治理与负责任AI研究者使用。
人工智能已广泛应用于医疗、金融等领域，通过自动化系统提升决策能力。然而，这些系统的可靠性高度依赖底层数据集的质量，引发对透明度、责任归属及潜在偏见的持续担忧。本文提出一种用于评估AI数据集开发的评分卡，聚焦系统卡片框架中的五个关键环节：数据字典、采集过程、数据构成、动机说明和预处理。该方法采用标准化录入表与评分标准，评估数据集的质量与完整性。在四个不同领域的数据集上应用后，揭示了各数据集的优势与改进空间，并通过评分体系生成针对性优化建议。评分卡兼顾技术性与伦理性，提供对数据实践的全面评估，旨在提升数据集的透明度与完整性。该方法为数据管理者和研究人员提供了切实可行的指导，有助于构建公平、可问责的决策支持系统。
原文摘要 · Abstract (English)
Artificial intelligence has transformed numerous industries, from healthcare to finance, enhancing decision-making through automated systems. However, the reliability of these systems is mainly dependent on the quality of the underlying datasets, raising ongoing concerns about transparency, accountability, and potential biases. This paper introduces a scorecard designed to evaluate the development of AI datasets, focusing on five key areas from the system card framework data development life cycle: data dictionary, collection process, composition, motivation, and pre-processing. The method follows a structured approach, using an intake form and scoring criteria to assess the quality and completeness of the data set. Applied to four diverse datasets, the methodology reveals strengths and improvement areas. The results are compiled using a scoring system that provides tailored recommendations to enhance the transparency and integrity of the data set. The scorecard addresses technical and ethical aspects, offering a holistic evaluation of data practices. This approach aims to improve the quality of the data set. It offers practical guidance to curators and researchers in developing responsible AI systems, ensuring fairness and accountability in decision support systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。