评估人机协作的准备度，而非单纯看模型准确率。
From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making
- 构建四维评估框架：结果、依赖行为、安全信号、学习过程。
- 通过交互记录衡量校准度、错误恢复与治理能力。
- 适合关注人机协同安全与可问责性的研究者与实践者。
人工智能系统正作为人类决策的合作者部署，但现有评估仍聚焦于模型准确率，而非人机团队是否具备安全有效的协作准备度。实证表明，许多失败源于依赖关系失调：当AI出错时过度依赖，当AI有用时却使用不足。本文提出以团队准备度为中心的人机决策评估框架，引入涵盖结果、依赖行为、安全信号和长期学习的四部分评价指标体系，并将其与人机协作的‘理解-控制-改进’（U-C-I）生命周期关联。通过操作化交互轨迹而非模型属性或主观信任报告，该框架支持对校准、错误恢复和治理能力的部署相关评估。旨在推动更可比的基准测试与累积性研究，促进更安全、更可问责的人机协作。
原文摘要 · Abstract (English)
Artificial intelligence (AI) systems are deployed as collaborators in human decision-making. Yet, evaluation practices focus primarily on model accuracy rather than whether human-AI teams are prepared to collaborate safely and effectively. Empirical evidence shows that many failures arise from miscalibrated reliance, including overuse when AI is wrong and underuse when it is helpful. This paper proposes a measurement framework for evaluating human-AI decision-making centered on team readiness. We introduce a four part taxonomy of evaluation metrics spanning outcomes, reliance behavior, safety signals, and learning over time, and connect these metrics to the Understand-Control-Improve (U-C-I) lifecycle of human-AI onboarding and collaboration. By operationalizing evaluation through interaction traces rather than model properties or self-reported trust, our framework enables deployment-relevant assessment of calibration, error recovery, and governance. We aim to support more comparable benchmarks and cumulative research on human-AI readiness, advancing safer and more accountable human-AI collaboration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。