开源AI评估库的8个月运维经验，揭示了评估系统建设的关键挑战与解决方案。
Developing and Maintaining an Open-Source Repository of AI Evaluations: Challenges and Insights
- 构建结构化贡献管理框架，支持70多个社区评估的规模化协作。
- 提出带不确定性量化的重采样与跨模型比较方法，提升评估可靠性。
- 建立系统化质控流程，保障评估结果可复现，适合评估开发者参考。
AI评估已成为衡量大语言模型能力与安全性的关键工具。本文基于对开源仓库$inspect_evals$长达八个月的维护经验，该库汇集了70余个社区贡献的AI评估。我们识别出实现与维护AI评估的核心挑战，并提出三项解决方案：(1) 结构化队列管理框架，用于扩展社区贡献；(2) 统计方法支持最优重采样与跨模型比较，并包含不确定性量化；(3) 系统性质量控制流程以确保可复现性。分析表明，AI评估需要超越传统软件开发的专用基础设施、统计严谨性及社区协同能力。
原文摘要 · Abstract (English)
AI evaluations have become critical tools for assessing large language model capabilities and safety. This paper presents practical insights from eight months of maintaining $inspect\_evals$, an open-source repository of 70+ community-contributed AI evaluations. We identify key challenges in implementing and maintaining AI evaluations and develop solutions including: (1) a structured cohort management framework for scaling community contributions, (2) statistical methodologies for optimal resampling and cross-model comparison with uncertainty quantification, and (3) systematic quality control processes for reproducibility. Our analysis reveals that AI evaluation requires specialized infrastructure, statistical rigor, and community coordination beyond traditional software development practices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。