分析了AI社会影响评估的现状，发现自评与第三方评估存在明显分工和盲区。
Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations
- 对比186份自评报告与248份第三方评估，揭示评估覆盖不均
- 自评报告在偏见与环境影响方面逐年减少，第三方更关注有害内容与性能差异
- 开发者掌握关键数据但常忽视披露，需政策强制透明
基础模型日益成为高风险AI系统的核心，治理框架依赖评估来衡量其风险与能力。尽管通用能力评估广泛开展，但涵盖偏见、公平性、隐私、环境成本和劳动力影响的社会影响评估仍不均衡。为描绘这一图景，我们首次对社会影响评估报告进行了全面分析，考察了186份第一方发布报告和248份第三方评估来源，并辅以开发者访谈。研究发现存在显著分工：第一方报告稀少且浅显，环境影响与偏见相关披露持续下降；而第三方评估在偏见、有害内容及性能差异方面覆盖更广、更严谨。然而，唯有开发者能权威披露数据溯源、内容审核劳动、成本与基础设施信息，但访谈显示这些披露仅在关联产品采纳或合规时才被重视。当前实践导致社会影响评估存在重大空白，亟需政策推动开发者透明化、强化独立评估生态，并建立共享基础设施以整合第三方评估结果。
原文摘要 · Abstract (English)
Foundation models are increasingly central to high-stakes AI systems, and governance frameworks now depend on evaluations to assess their risks and capabilities. Although general capability evaluations are widespread, social impact assessments covering bias, fairness, privacy, environmental costs, and labor remain uneven. To characterize this landscape, we conduct the first comprehensive analysis of social impact evaluation reporting, examining 186 first-party release reports and 248 third-party evaluation sources, supplemented by developer interviews. We find a stark division of labor: first-party reporting is sparse, often superficial, and declining in areas like environmental impact and bias, while third-party evaluators provide broader, more rigorous coverage of bias, harmful content, and performance disparities. However, only developers can authoritatively report on data provenance, content moderation labor, costs, and infrastructure, yet interviews reveal these disclosures are deprioritized unless tied to product adoption or compliance. Current practices leave major gaps in assessing societal impacts, underscoring the need for policies that mandate developer transparency, strengthen independent evaluation ecosystems, and create shared infrastructure for aggregating third-party evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。