提出交互就绪框架,评估AI在人类角色中的行为表现。
Interaction Readiness: A Framework for Building and Evaluating AI Agents in Human Roles
- 区分内容与交互规范,明确角色行为要求
- 发现内容准确与交互质量独立,权威失准是主要问题
- 提供可落地的规范模板与部署前后审计流程
构建承担角色的AI代理的产品与工程团队面临评估空白:代理可能输出准确、安全且流畅的内容,但仍无法满足其角色的行为要求。本文提出‘交互就绪’框架,用于定义和评估这一缺失的表现层。该框架将内容规范(决定代理的知识与言辞)与交互规范(定义代理在角色化交流中的行为方式)分离。交互规范要求团队在部署前明确角色目的、权限边界、常见情境、边界案例、修复行为及审计标准。通过四个代理操作实现:理解目的、校准权限、管理语气、修复故障。基于StudyChat公开数据集(学生与AI辅导代理的互动)的实证表明,内容准确性与交互质量是两个独立维度:代理可能事实正确但辅导无效,或交互得体但技术错误。最普遍的问题是权限校准失败——代理知道如何回答,却不知是否、何时、以何种方式能以导师身份作答。论文将研究结果转化为可应用的规范模板与审计流程,供产品与工程团队在部署前与后使用。
原文摘要 · Abstract (English)
Product and engineering teams building role-bearing AI agents face an evaluation gap: an agent can produce accurate, safe, and fluent content while still failing the behavioral requirements of its assigned role. This paper introduces Interaction Readiness as a framework for specifying and evaluating that missing layer of performance. The framework separates content specifications, which govern what an agent knows and says, from interaction specifications, which define how an agent should conduct itself in a role-governed exchange. Interaction specifications require teams to define role purpose, authority boundaries, recurring situations, boundary cases, repair behaviors, and audit criteria before deployment. We operationalize interaction readiness through four agent operations: understanding purpose, calibrating authority, managing tone, and repairing breakdowns. Using StudyChat, a public dataset of student interactions with an AI tutoring agent, we show that content accuracy and interaction quality are independent dimensions: an agent may be factually correct while failing as a tutor, or interactionally sound while technically wrong. The most persistent failure is authority miscalibration: the agent often knows how to answer, but not whether, when, or how the tutor role permits it to answer. The paper translates these findings into a specification template and audit procedures that product and engineering teams can apply before and after deployment
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。