现有AI评估方法难测机器是否真像人，论文指出了三大缺陷并给出改进方案。
On Benchmarking Human-Like Intelligence in Machines
- 用真人评测九个主流AI基准，发现标签、任务设计不靠谱
- 指出当前评测缺乏人类真实反应的多样性和不确定性
- 提出五条新建议，让未来评估更贴近人类认知能力
近期人工智能发展催生了强大的计算模型，这些模型通过学习海量人类生成数据，被视作人类认知的近似模拟。然而我们指出，当前多数AI评估范式不足以衡量模型的人类认知能力。主要问题包括：缺乏人类验证的标签、未能充分反映人类反应的变异性与不确定性、依赖简化且生态无效的任务。通过对九个现有AI基准进行人类评估研究，我们揭示了任务设计与标签体系的重大局限性。为此，本文提出五项具体改进建议，以推动未来AI评估更严谨、更具意义，从而真正理解机器在多大程度上具备人类认知特征。
原文摘要 · Abstract (English)
Recent advances in Artificial Intelligence (AI) have yielded powerful computational models that, by learning from vast amounts of human-generated data, are increasingly posited as approximate models of human cognition. However, we argue that many current evaluation paradigms for AI are insufficient for assessing human-like cognitive capabilities in these models. We identify a set of key shortcomings: a lack of human-validated labels, inadequate representation of human response variability and uncertainty, and reliance on simplified and ecologically invalid tasks. We support our claims by conducting a human evaluation study on nine existing AI benchmarks, suggesting major limitations in task and label designs. To address these limitations, we propose five concrete recommendations for future AI evaluation efforts that will enable more rigorous and meaningful understanding of human-like cognitive capacities in AI models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。