提出五维评估卡SPHERE，规范人机协作系统评价标准
SPHERE: An Evaluation Card for Human-AI Systems
- 设计五维度评估框架，覆盖评估对象、方法、参与者等
- 分析39个系统发现当前评估缺乏透明度与一致性
- 适合研究者与开发者用于改进人机系统评估设计
大语言模型时代，多样化人机交互系统的有效评估方法日益复杂。为促进更透明的文档记录并推动对人机系统评估设计的讨论，本文提出评估卡SPHERE，涵盖五个关键维度：1）评估什么？2）如何评估？3）谁参与评估？4）何时评估？5）如何验证评估？我们基于SPHERE对39个真实人机系统进行了评审，梳理了当前评估实践现状及改进空间，并提出三项提升评估有效性与严谨性的建议。
原文摘要 · Abstract (English)
In the era of Large Language Models (LLMs), establishing effective evaluation methods and standards for diverse human-AI interaction systems is increasingly challenging. To encourage more transparent documentation and facilitate discussion on human-AI system evaluation design options, we present an evaluation card SPHERE, which encompasses five key dimensions: 1) What is being evaluated?; 2) How is the evaluation conducted?; 3) Who is participating in the evaluation?; 4) When is evaluation conducted?; 5) How is evaluation validated? We conduct a review of 39 human-AI systems using SPHERE, outlining current evaluation practices and areas for improvement. We provide three recommendations for improving the validity and rigor of evaluation practices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。