arXiv:2603.00063cs.CYcs.AI2026-03被引 2

为AI能力测量建立科学框架,区分能力与表现。

Measuring What AI Systems Might Do: Towards A Measurement Science in AI

  • 将AI能力视为因果条件下的行为倾向,需通过反事实推理验证
  • 现有评估忽略因果变量,无法真实测量系统倾向
  • 适合关注AI安全与可解释性的研究者和政策制定者

科学家、政策制定者、企业领袖和公众关心现代人工智能系统可能做什么。然而,能力、倾向、技能、价值观和能力等术语常被混用,并与可观测性能混淆,现有AI评估很少明确说明所测量的量。我们主张,能力和倾向是处置性属性——由情境条件与行为输出之间的反事实关系定义的系统稳定特征。测量一种处置性属性需要:(i) 假设哪些情境属性具有因果相关性;(ii) 独立操作化并测量这些属性;(iii) 实证映射这些属性的变化如何影响行为发生的概率。主流的AI评估方法,从基准平均值到基于数据的潜在变量模型(如项目反应理论),完全跳过了这些步骤。基于科学哲学、测量理论和认知科学的思想,我们构建了将AI能力与倾向视为处置性的严谨框架,阐明现有评估为何无法测量它们,并提出符合处置性原则的科学可靠评估应具备的要素。

原文摘要 · Abstract (English)

Scientists, policy-makers, business leaders, and members of the public care about what modern artificial intelligence systems are disposed to do. Yet terms such as capabilities, propensities, skills, values, and abilities are routinely used interchangeably and conflated with observable performance, with AI evaluation practices rarely specifying what quantity they purport to measure. We argue that capabilities and propensities are dispositional properties - stable features of systems characterised by counterfactual relationships between contextual conditions and behavioural outputs. Measuring a disposition requires (i) hypothesising which contextual properties are causally relevant, (ii) independently operationalising and measuring those properties, and (iii) empirically mapping how variation in those properties affects the probability of the behaviour. Dominant approaches to AI evaluation, from benchmark averages to data-driven latent-variable models such as Item Response Theory, bypass these steps entirely. Building on ideas from philosophy of science, measurement theory, and cognitive science, we develop a principled account of AI capabilities and propensities as dispositions, show why prevailing evaluation practices fail to measure them, and outline what disposition-respecting, scientifically defensible AI evaluation would require.

AI评估测量科学处置性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。